Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Subtask] support Audit Log framework #3973

Closed
Tracked by #4887
hanwxx opened this issue Jun 26, 2024 · 4 comments
Closed
Tracked by #4887

[Subtask] support Audit Log framework #3973

hanwxx opened this issue Jun 26, 2024 · 4 comments
Assignees
@hanwxx
Labels
0.7.0 Release v0.7.0 improvement Improvements on everything
Milestone
Gravitino 0.7.0

Comments

@hanwxx
Copy link
Contributor

hanwxx commented Jun 26, 2024 •
edited
Loading

What would you like to be improved?

Based on the metadata change message, in our company has following consumption scenarios:

  1. Data governance: based on job lineage, metadata change messages will be sent to affected job owners, so they can decide whether to adjust jobs
  2. Metadata search: After processing, the medata data will be imported into ES to provide complex search capabilities.
  3. Audit scenario: the change operation is stored persistently for audit.

How should we improve?

So i want to add a audit listenr, listening to the change event, then send the message to the external MQ ( like RocketMQ）
No response
@hanwxx hanwxx added the improvement Improvements on everything label Jun 26, 2024
@FANNG1
Copy link
Contributor

FANNG1 commented Jun 27, 2024

seems reasonable to add a event consumer to deliver events to message queues, Besides this, could event system satisfy your requirement?

@hanwxx
Copy link
Contributor Author

hanwxx commented Jul 1, 2024

Get it，event system can meet the requirement

@jerryshao
Copy link
Contributor

jerryshao commented Jul 2, 2024 •
edited
Loading

It would be nice to have an audit log framework in Gravitino, like to define the format of audit log, the necessary information of each audit record. Besides, I think we can make the storage of audit log configurable, user can implement the specific storage like rocket mq, files, etc. For Gravitino, we don't have to bind to one storage.

@hanwxx
Copy link
Contributor Author

hanwxx commented Jul 11, 2024

This is design document. Please review it
https://docs.google.com/document/d/1qyphXC9xLsg2mcKhHLxqbCudUqeEpCZDKvQo-ztY0HE/edit?usp=sharing

@jerryshao jerryshao added this to the Gravitino 0.7.0 milestone Aug 6, 2024
@hanwxx hanwxx mentioned this issue Aug 19, 2024
Merged
@hanwxx hanwxx changed the title [Improvement] Add new event bus listener for audit [Subtask] Design Audit Log framework Sep 9, 2024
@hanwxx hanwxx mentioned this issue Sep 9, 2024
Closed
2 tasks
@FANNG1 FANNG1 closed this as completed in dffff6b Oct 21, 2024
@FANNG1 FANNG1 added the 0.7.0 Release v0.7.0 label Oct 21, 2024
@FANNG1 FANNG1 changed the title [Subtask] Design Audit Log framework [Subtask] support Audit Log framework Oct 21, 2024
mplmoknijb pushed a commit to mplmoknijb/gravitino that referenced this issue Nov 6, 2024
@hanwxx @yuqi1129 @mplmoknijb
[apache#3973] improvement(core):support audit log (apache#4575)
fd23d0c 
### What changes were proposed in this pull request?

Design a gravitino audit framework.

### Why are the changes needed?

fixes: apache#3973

### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
1. Add UT

---------

Co-authored-by: han <hanwangxin@xiaomi.com>
Co-authored-by: Qi Yu <yuqi@datastrato.com>
@FANNG1 FANNG1 mentioned this issue Nov 12, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hanwxx hanwxx

Labels
0.7.0 Release v0.7.0 improvement Improvements on everything
Projects
None yet
Milestone
Gravitino 0.7.0
Development

No branches or pull requests
3 participants
@jerryshao @FANNG1 @hanwxx