[Improvement] Add new event bus listener for audit

[hanwxx]

What would you like to be improved?

Based on the metadata change message, in our company has following consumption scenarios:

1. Data governance: based on job lineage, metadata change messages will be sent to affected job owners, so they can decide whether to adjust jobs
2. Metadata search: After processing, the medata data will be imported into ES to provide complex search capabilities.
3. Audit scenario: the change operation is stored persistently for audit.

How should we improve?

So i want to add a audit listenr, listening to the change event, then send the message to the external MQ ( like RocketMQ）
No response

[FANNG1]

seems reasonable to add a event consumer to deliver events to message queues, Besides this, could event system satisfy your requirement?

[hanwxx]

Get it，event system can meet the requirement

[jerryshao]

It would be nice to have an audit log framework in Gravitino, like to define the format of audit log, the necessary information of each audit record. Besides, I think we can make the storage of audit log configurable, user can implement the specific storage like rocket mq, files, etc. For Gravitino, we don't have to bind to one storage.

[hanwxx]

This is design document. Please review it
https://docs.google.com/document/d/1qyphXC9xLsg2mcKhHLxqbCudUqeEpCZDKvQo-ztY0HE/edit?usp=sharing