Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better document pipeline concepts. #589

Open
bangerth opened this issue May 16, 2024 · 3 comments
Open

Better document pipeline concepts. #589

bangerth opened this issue May 16, 2024 · 3 comments
Labels
help wanted Extra attention is needed

Comments

@bangerth
Copy link
Contributor

I've been reading my way through the documentation of the pipeline classes and noticed two things:

  • All of the examples shown always use serial stages. It would be nice to have an example or two where a stage is parallel.
  • TaskFlow uses a concept of "lines" in pipelines, but the truth is that I haven't quite been able to figure out what that means. I think it replicates the whole pipeline multiple times to route items/tokens down one or the other. But in truth I don't quite understand, in particular because the pipeline graphs are all meshes that also have dependencies between lines -- though there too I'm unsure because the graph doesn't make clear whether the x-axis represents lines and the y-axis the stages of a pipeline, or the other way around. It would be nice to see the concept of a "line" documented somewhere.
  • TaskFlow uses somewhat unusual terminology in that the places in a pipeline where work is done are called "pipes", whereas everyone else thinks of these places as the "nodes" of a graph and the "pipes" are just the connecting edges that transport data from one node to another. That's also how the task graphs are visualized. But in TaskFlow, the "pipes" are the nodes of the graph, not the edges. All of that is fine -- it's just terminology -- but it would be nice to explain these concepts in a central place.
@tsung-wei-huang
Copy link
Member

Hi @bangerth thank you for your suggestions. I will take that in mind to improve the doc. When we design the pipeline, we didn't know how to exactly match each pipe to a task, and that's why we tried not to mix those terms (pipe, node) together. In fact, the underlying implementation is based on condition-task and composition, but this of course should be transparent to users.

@bangerth
Copy link
Contributor Author

@tsung-wei-huang I understand.

Would you mind explaining (or linking to a description) of what a "line" is in a pipeline? I could not figure that out from the documentation or the examples.

@tsung-wei-huang
Copy link
Member

Hi @bangerth , line means the parallelism of your pipeline program. If you have four lines, then taskflow will use four tasks to build your pipeline structure, which goes to a parallelism of 4 threads. The scheduling runs in a circular fashion like the first figure here

@tsung-wei-huang tsung-wei-huang added the help wanted Extra attention is needed label May 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants