Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Communication broker - part 1 / Multi-application handling interface part-2 #49

Merged
merged 16 commits into from
May 10, 2023

Conversation

YooSunYoung
Copy link
Member

@YooSunYoung YooSunYoung commented Apr 26, 2023

I fixed some problems from #45 thanks to the communication protocol updates.

Related to #41, #43

Main changes

  • Communication channel mapping per application is now done by the communication broker, beamlime.communications.broker.CommunicationBroker.
  • broker follows the BeamlimeBrokerProtocol which inherits CommunicationProtocol.
  • broker will get, put, consume and produce data according to the type of channel that application is subscribing to.
  • data-stream configuration now has all application specs including data-reduction.
  • __init__ function of BeamlimeApplicationInterface updated so that it does not need to have config as a dictionary and parse it. All specs can be dumped by **config when it's instantiated.
  • You can test this with the src/beamlime//offine/offline_system_example.ipynb. It will be moved to docs later...!

TODO - Things I will continue after the vacation!

  • put and poll method implementation.
  • Add tests for communication interfaces.
  • Add communication handles. (Currently single-process queue only. We need a blackboard, multi-processing queue and kafka, kafka interfaces are already tested but still need to implement auto-configuration part.)
  • Figure out what issues we will have in multi-processing, since now all application is sharing one broker instance. (I have an idea of instance twins...!)
  • Add logging into broker and data flow monitor.

@YooSunYoung
Copy link
Member Author

I will merge this one since it was reviewed in other PR and will be reviewed again in the PR #53

@YooSunYoung YooSunYoung merged commit efe782e into main May 10, 2023
@YooSunYoung YooSunYoung deleted the feature-communication branch May 10, 2023 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant