Road to 1.0.0

[adamdubiel]

I think it is time we think about 1.0.0 release. At this point i think Hermes is feature complete and those features that are yet in incubation will surely mature soon, thanks to our early adopters.

However, there are a few major, possibly breaking things that need to be done before we move into 1.x.x space in mostly non-functional area:

• drop support for Kafka 0.8.x protocol and move to new implementation of Consumers - must have for 0.9.0 release
• new, process-based implementation of Consumers already reviewed in 467 assignment executor blocked #505
• drop servlets abstraction in Frontend, also already in progress
• create consistent naming convention for properties for all modules: Frontend, Management and Consumers and rename exisiting properties to match new convention (with some migration script to automate) (i.e. some properties include time unit like .seconds or .millis, others don't)
• change modules & extensions approach - i.e. activating elasticsearch module should not require to build a new project and plug all things programmatically; we could probably leverage some classpath scanning - to discuss
• choose one DI framework for all modules (including Management) - i think Dagger2 or Guice; hk2 is not fit for such a big project
• reconsider using one tech stack for all modules; Management is too different from Consumers and Frontend which makes reusing components & documentation painful.

The list is open for discussion. I would especially appreciate opinion from @sslavic and @gamefundas and @hikrrish as our known clients from outside Allegro. Of course @allegro/hermes is welcome to comment as well.

[adamdubiel]

One more thing:

• review and create consistent logging format which could be easily searchable (i.e. for all subscription relate logs: [{subscriptionName}] message)

[wendigo]

Nice roadmap 👍

[sslavic]

Sounds good to me 👍

[gamefundas]

The core enhancement list looks good to me. Few other wish-list items from my side. Some are debatable and more of 1.x features.

1. Hermes Management + Console – Perhaps merge features in to one module, Setup topics, subscriptions, view/backup, message history, view message content, replay etc
2. Hermes monitoring, Key performance indicators, Stats, Metrics of Frontend and Consumer – Real-time view in Grafana and historical data pushed to ES for analysis.
3. Documentation - PDF documentation in more detailed than online one, Examples are outdated especially with payload attributes
4. Pluggable architecture for Tracing – ES configuration, Enabled/Disable
5. Docker images of each module for quick production cluster setup
6. More documentation and examples around Avro support. Couldn't get our example working.
7. Acknowledgement message back to Producer – Support for registering Http callback which can be invoked on successful message delivery with the client's response payload - Saw your response on the other thread Receiving Payload/ ACK from clients in consumer module  #522 so will explore it further. But I guess this will be a nice feature to have when the producer wants to ensure guaranteed delivery and accept some sort of correlation id back besides the default Http 200 Ok.
8. Support pluggable authentication for Clients – Http Basic Auth (cleaner way than sending user/password in URL), Kerberos etc. Can't think of a good way to do this though.

These are few things I had in mind. Please pardon me if some are already captured as issues or they don't fit in the overall Hermes vision.

[adamdubiel]

Great, thanks for this list :) I will take it into consideration when planning new stuff in Hermes. As of today:

3. We will fix this, as we said yesterday.
4. It's already here :) https://hub.docker.com/r/allegro/hermes-frontend/
5. This would be turning push into pull, for now i don't think it will happen.
6. It's on the way, better Basic Auth and OAuth2 will arrive this month.

[gamefundas]

Another configurable option for elasticsearch feature would be to store the message payload along with Hermes message id. We have seen that it kind of helps when looking for any lost messages that needs to be replayed etc.

[adamdubiel]

Okay - but only for discarded ones?

[gamefundas]

I was thinking in general as in cases where the payload is small the client can choose to store the payload in ES and it serves as trace/log for any triage. Especially answer cases where the subscriber complains I didnt recieve it.

[adamdubiel]

Tracing info is a debugging tool - we store everything necessary to recover body of a message either from Kafka (partition & offset) or Hadoop (message id - if Kafka is mirrored to Hadoop). When "i didn't receive it" clients come, we show him tracing logs, as the body usually doesn't matter so much, and this is enough in most cases. I'm a bit concerned about putting all traffic from Kafka to Elastic.

Currently for the discarded events Hermes has an endpoint where you can fetch the message from Kafka using its partition & offset. It also included in Hermes Console: when you have a list of last 100 discarded messages there is a magnifier icon. Click it to get the event body retrieved from Kafka.

[gamefundas]

When I think of this more, we only need the payloads for discarded messages. The other delivered messages if needs to be replayed can always use the Hermes message id. We are not using the Console for now but if there is a way we can get all discarded message id's that should be enough, the source can be requested to publish then again.

[adamdubiel]

Every action available via Console is of course a REST API call, so it's all there. However our endpoints only show sth like last 100 discarded messages and we usually just go straight to Elastic for more data.

Maybe you could create a separate issue where we can discuss this topic?

[gamefundas]

I noticed the waffle board is gone. Any place to look forward to what's coming up?

[druminski]

Currently we have version 1.3.0.