Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Proof of concept: reliable asynchronous processing #103
Asynchronous processing should be possible for long-running functions.
Or via Header:
Nice to have:
Have looked into Kafka - design looks overly complex for task at hand.
Amazing ! You roks ! :)
I used FaaS to deploy some functions that my machine learning experts made with magic.
FaaS works great for prediction service as it is a synchronous task.
As some processing functions need time (training our models), we would use async processing and we already have a workflow ready for testing.
When our developments will gain in stability, we will be able to communicate and mention FaaS as the solution we decided to use in a tweet or some presentations we will make ('cause we are working with French industries in aerospace
It would probably help to map out the use cases for the async processing first as there are a couple different ones I can think of that usually require different guarantees and metrics. anyone know what the users of this library would favor in use cases for this as this would ease up choosing the right queueing options here too. kafka for example might seem overly complex (and it is complex) but it has its uses, but usually I wouldnt choose that for simple response queues like here generally. nats is nice general purpose, but i fear doesnt encompass all options. there is also the possibility of a mixed solution making it simple pub/sub with separate log database (usually you need recent stuff, which is in mem, but you dont lose old stuff this way and support stuff like "oh my car is offline for 15min because of shit internet, but can still get its response"), which gives quite a lot of flexibility and isnt that hard to implement.
As you have mentioned - the various queue implementations available have their own pros/cons. Ideally it should be easy to swap between different "queue" providers or implementations.
This initial branch / work is based around a NATs streaming queue which does have persistence and resilience.
You can see the progress here:
Guide to testing the branch:
referenced this issue
Aug 18, 2017
Work merged into master and released in 0.6.3 https://github.com/alexellis/faas/releases/tag/0.6.2
If anyone wants to start on a Kafka implementation that would be great - otherwise let's spend time using the async code and doing edge-case testing.