Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove AsyncPipeline #793

Closed
norberttech opened this issue Nov 14, 2023 · 0 comments · Fixed by #794
Closed

Remove AsyncPipeline #793

norberttech opened this issue Nov 14, 2023 · 0 comments · Fixed by #794
Assignees
Milestone

Comments

@norberttech
Copy link
Member

Our current implementation of asynchronous processing is built on top of React/AMPhp. Flow is simply spawning new processes and it's sending the pipeline into it, later those processes are receiving rows over TCP, transforming them and sending back also through TCP to the main process.

This approach, even that is pretty cool is not brining enough value in order to justify the maintenance around it.
The main problem comes from the fact that processing is not the biggest bottleneck, our main problem comes from extraction, which is not included in our current async processing model.

Another problem is that async processing is currently impossible through phar/docker.

For now we are going to remove the AsyncPipeline and react/amphp adapters, which should reduce the complicity of our codebase allowing us to easier refactor Pipelines (we are still missing Executor). One that is done, we should look again into async processing, however this time we need to start from async extraction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant