Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[INLONG-11145][Agent] Optimize the logic of supplementing data #11146

Merged
merged 1 commit into from
Sep 19, 2024

Conversation

justinwwhuang
Copy link
Contributor

@justinwwhuang justinwwhuang commented Sep 19, 2024

Fixes #11145

Motivation

The logic for supplementing data needs to be optimized to prevent data loss

Modifications

  1. If the action queue is full, the action cannot be directly discarded
  2. Determination of completion of supplementary data task: The number of scanned files is equal to the number of collected files

Verifying this change

(Please pick either of the following options)

  • This change is a trivial rework/code cleanup without any test coverage.

  • This change is already covered by existing tests, such as:
    (please describe tests)

  • This change added tests and can be verified as follows:

    (example:)

    • Added integration tests for end-to-end deployment with large payloads (10MB)
    • Extended integration test for recovery after broker failure

Documentation

No doc needed

@justinwwhuang justinwwhuang merged commit 73f477a into apache:master Sep 19, 2024
11 checks passed
Zkplo pushed a commit to Zkplo/inlong that referenced this pull request Sep 21, 2024
@justinwwhuang justinwwhuang deleted the b11145 branch September 23, 2024 02:17
PeterZh6 pushed a commit to PeterZh6/inlong that referenced this pull request Sep 23, 2024
wohainilaodou pushed a commit to wohainilaodou/inlong that referenced this pull request Oct 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Improve][Agent] Data supplementation needs optimization
3 participants