Skip to content

[INLONG-4178][Manager] Refactor getSortSource API for Sort Standalone#4917

Merged
dockerzhang merged 6 commits intoapache:masterfrom
vernedeng:INLONG-4178
Jul 11, 2022
Merged

[INLONG-4178][Manager] Refactor getSortSource API for Sort Standalone#4917
dockerzhang merged 6 commits intoapache:masterfrom
vernedeng:INLONG-4178

Conversation

@vernedeng
Copy link
Copy Markdown
Contributor

@vernedeng vernedeng commented Jul 7, 2022

Prepare a Pull Request

(Change the title refer to the following example)

Motivation

In order to support new db model and reduce the number of query to database, the interface of sort sdk config management should be refactored.

One cluster can register multiple tasks, each task represents one flow from mq to downstream sink, and each task can register multiple InlongGroupId.

Modifications

Use local cache to maintain SortSourceConfigResponse of all tasks.
Reload all configs each 10 seconds.

Main step:

  1. Get all streams, and group them to by cluster and task
  2. Get all inlong groups, and group them by groupId
  3. Get all clusters, and group by cluster tag
  4. Prepare CacheZones for each cluster and task

Verifying this change

(Please pick either of the following options)

  • This change added tests and can be verified as follows:

    • Empty request params
    • Wrong request params
    • Same md5
    • Corret request params

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? JavaDocs

@vernedeng vernedeng changed the title [INLONG-4178] Refactor getSortSource API for Sort Standalone [INLONG-4178][Manager] Refactor getSortSource API for Sort Standalone Jul 7, 2022
Copy link
Copy Markdown
Contributor

@luchunliang luchunliang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

@dockerzhang dockerzhang merged commit cfaba41 into apache:master Jul 11, 2022
@vernedeng vernedeng deleted the INLONG-4178 branch May 26, 2023 03:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature][Manager] Refactor getSortSource interface

5 participants