Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improve][DataProxy] Metadata synchronization management optimization #8758

Closed
2 tasks done
gosonzhang opened this issue Aug 17, 2023 · 0 comments · Fixed by #8759
Closed
2 tasks done

[Improve][DataProxy] Metadata synchronization management optimization #8758

gosonzhang opened this issue Aug 17, 2023 · 0 comments · Fixed by #8759

Comments

@gosonzhang
Copy link
Contributor

Description

In the running environment, there are the following problems that need to optimize the metadata management of DataProxy:

  1. When DataProxy synchronizes metadata from Manager, it may take too long to pull data. We add a pull time consumption alarm setting to remind when encountering such an abnormal situation during use;
  2. When DataProxy synchronizes metadata from the Manager and loads data from the metadata, it may be out of sync. We need to improve to avoid the situation where the metadata that is dropped to the file has not had time to load and take effect;
  3. For the metadata synchronized by DataProxy, if the topic configuration is deleted but the message has already entered the channel from the source and is waiting to be sent, this part of the data should continue to be sent instead of being discarded.

InLong Component

InLong DataProxy

Are you willing to submit PR?

  • Yes, I am willing to submit a PR!

Code of Conduct

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant