[Discussion] Revisit RocksDB and ZeebeDb responsibility

[Zelldon]

Description

Last year we implemented the engine abstraction to support our team split. During implementing the abstraction and planning we have decided to assign ZDP as responsible team. . There have been now made several changes to the API because it was needed for ZPA. Some configurations might need to be changed for better access and performance, etc.

There is coming up a new pressing topic #12033 which is highly related to ZeebeDB, RocksDB and how they are accessed, and what type of data we store and how.

As we (ZDP) have no control over the data, @npepinpe was questioning me again whether this makes sense and I kind of have to agree.

Some points which stands out:

• ZPA uses ZeebeDB in order to store there data
• ZPA uses ZeebeDB in order to receive there data
• ZDP has almost zero knowledge what type of data is stored and how
• ZDP is now responsible for the ZeebeDB API
• ZDP is right now in charge of RocksDB configuration

I/We see here some conflicts/challenges, which might be resolved when ZeebeDB is in ZPA's hands. This would allow the ZPA team to design the API to their needs without conflicting with us, configuring the underlying database to their needs. Furthermore, it might make it even easier to replace RocksDB with something completely different which maybe full-fils better there needs, like SQLite.

I think this is something we should discuss and think about again, especially you both @megglos and @abbasadel since this would allow each team to focus more on certain things and reduce conflict potential.

BTW there might be some arguments that performance is the mission of ZDP. I have to say that this can't be a mission of just one team. This needs to be a target of the whole platform, otherwise, it will not work.

[npepinpe]

We have to change the gazintas and the gazoutas ;)

But really, it's very hard for us to optimize properly when we don't own the data model. The example of evaluating SQLite vs RocksDB for example is whether it could be more efficient to use a relational data model (and a technology which is optimized to understand it) instead of us hacking relationships on top of RocksDB. It also highlights that the ZDP team is hard pressed to make a proper comparison there since we aren't super familiar with the model itself, or how it's accessed, making it hard to intelligently optimize things like how much work is done in memory and how much is offloaded to disk.

Also just sayin, but my original vision for the team split was always to have RocksDB/zb-db owned by the ZPA team 👀

EDIT: BTW, just to clarify, the SQLite thing is an example. I doubt it would be the right thing long term, but its just to illustrate the point.

[korthout]

@abbasadel Please co-ordinate with @megglos :)

[korthout]

Thanks @megglos, I'd like to rephrase my statements:

• I wondered whether ZDP's goal is to provide a platform that can be used to build any replicated and fault tolerant application, and not just Zeebe
  • If so, then I expect that any other application would also need a way to keep track of the state in their state machine. So it would be useful if the platform provides some solution for this. In practice, it doesn't matter what this looks like (in the past it was just in memory, later it became RocksDB to avoid running out of memory), so I could even imagine some API that allows connecting different data stores.
  • If not, then I think it makes sense to look at how the team can benefit the most. In that case, I think it makes sense to shift the responsibility over to ZPA, as we use it and have specific needs for our data store.
  • Additionally, I don't think it makes sense that ZDP is focusing on performance while ZPA is not, but that may be a different discussion.

[Zelldon]

Any status update here @abbasadel @megglos ?