You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Our current implementation of asynchronous processing is built on top of React/AMPhp. Flow is simply spawning new processes and it's sending the pipeline into it, later those processes are receiving rows over TCP, transforming them and sending back also through TCP to the main process.
This approach, even that is pretty cool is not brining enough value in order to justify the maintenance around it.
The main problem comes from the fact that processing is not the biggest bottleneck, our main problem comes from extraction, which is not included in our current async processing model.
Another problem is that async processing is currently impossible through phar/docker.
For now we are going to remove the AsyncPipeline and react/amphp adapters, which should reduce the complicity of our codebase allowing us to easier refactor Pipelines (we are still missing Executor). One that is done, we should look again into async processing, however this time we need to start from async extraction.
The text was updated successfully, but these errors were encountered:
Our current implementation of asynchronous processing is built on top of React/AMPhp. Flow is simply spawning new processes and it's sending the pipeline into it, later those processes are receiving rows over TCP, transforming them and sending back also through TCP to the main process.
This approach, even that is pretty cool is not brining enough value in order to justify the maintenance around it.
The main problem comes from the fact that processing is not the biggest bottleneck, our main problem comes from extraction, which is not included in our current async processing model.
Another problem is that async processing is currently impossible through phar/docker.
For now we are going to remove the AsyncPipeline and react/amphp adapters, which should reduce the complicity of our codebase allowing us to easier refactor Pipelines (we are still missing Executor). One that is done, we should look again into async processing, however this time we need to start from async extraction.
The text was updated successfully, but these errors were encountered: