Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Feedback needed #48
Hello fellow Gophers!
I apologise as this project slipped out of my scope for several years.
I still have some ideas and plans of maintaining it, but I need some feedback on how it is going to be used by people who actually tried it. So, I would really appreciate your answers to the following questions in this thread or in any other form (e.g. via email, please find the address in my profile).
Why this is important
As you might have noticed, this codebase is a bit dated. In fact, it was written in 2011 and didn't change much ever since. My own views on how an FBP library should work have changed over time. So, I think this library deserves a rewrite.
My views can be similar or different from yours, while I'm not building this library only for myself. That's why feedback is appreciated so much.
Thank you for participating!
First of all, a big thank-you for contributing this to the community.
I have actually left the company where I worked on this, but it was with @tonygallagher and @lrgar. They might have more to add. I seem to recall an issue connecting multiple components to an exit port. We got around this by adding an aggregation component before the graph exit.
Hi @trustmaster, I want to also thank you so much for contributing this!
I've had a lot of fun by playing with GoFlow for bioinformatics use cases, and also learned a lot both about Go and some FBP, by studying it.
To answer the questions one at a time:
I've did experimentation with bioinformatics components using it (like this). I eventually got worried about performance of reflection though, and has since explored a way to build FBP-like programs using only plain channels (this is ongoing work in my scipipe and flowbase libraries).
Prefer to work mainly in text form. I feel graphs can be a very useful addition for sketching at design time, and for presentation though.
I've used JPM's fbpdraw a bit for presentation purposes. I like that it is simple and just works. Haven't had time / patience for setting up any more complicated tools.
I prefer a central
In our work, pipelines runs have had a clear start and finish time, so on-demand stop and start has not been something we saw a need for.
I like that it is Go code, as that enables re-use of Go-tooling. I also found the API natural and easy to to understand and work with.
I'm worried about the performance hit from reflection though for data intensive pipelines. If there is any way to avoid reflection on each data read, e.g. by just using reflection to do set up channels, which are then used for the data communication, that would be great.
Keep up the great work!
@samuell regarding your concern about the use of reflection, in the latest version it's mostly used to wire up the channels. The only thing that is used on each data read is passing the arbitrary data as
Ah, interesting, I should have another deep look at the code!
Fist of all: Great work! It's much fun to use the package!
What kind of application did you use GoFlow for? (E.g. bioinformatics, ETL, web backend, IoT, etc.)
Did you use it for personal, business, or academic purposes?
Do you prefer working with graphs in visual or text form?
Which visual tools have you used and which ones do you prefer?
Do you prefer a Component to have main Loop(), or do you prefer setting up handler functions on each port (OnPortName())?
Do you prefer processes to stay resident in memory or started and stopped on demand?
Have you ever used State Locks, Synchronous Mode, Worker Pool, or other advanced features of GoFlow?
Please tell me what you liked about GoFlow and what you would like to be added or changed.
For my part, it would be much more for an advanced ETL/node-based pipeline. It would get lots of traction wiht DevOps as it could made easier and better to aggregate data (apis, logs) or to process data to import with dynamic/composable pipelines, chained by event triggers or conditions. It would also awesome for building smart/refined datasets for AI and Deep Learning.
Example by video