Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[INLONG-8242][CI] Separate the distribution module to improve the performance of check pipeline #8243

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[INLONG-8242][CI] Separate the distribution module to improve the performance of check pipeline #8243

wants to merge 2 commits into from

Conversation

featzhang
Copy link
Member

@featzhang featzhang commented Jun 14, 2023 •
edited

The inlong-distribution module uses assembly to compress and package the output of each module together. However, the module dependencies are not clearly specified in the pom file, causing maven to infer dependencies. The inlong-distribution module will cause parallel packaging failure and affect the performance of the CR pipeline, so this module is separated out.
When executing a CR pipeline, no packaging is required. Other modules can be compiled in parallel.

mvn clean package -B -T1C -Pfast
[INFO] Reactor Summary for Apache InLong 1.8.0-SNAPSHOT:
[INFO]
[INFO] Apache InLong ...................................... SUCCESS [  4.956 s]
[INFO] Apache InLong - Common ............................. SUCCESS [ 40.388 s]
[INFO] Apache InLong - SDK ................................ SUCCESS [  9.207 s]
[INFO] Apache InLong - SDK Common ......................... SUCCESS [ 22.131 s]
[INFO] Apache InLong - TubeMQ ............................. SUCCESS [  2.918 s]
[INFO] Apache InLong - TubeMQ Core ........................ SUCCESS [ 59.387 s]
[INFO] Apache InLong - TubeMQ Client ...................... SUCCESS [ 10.190 s]
[INFO] Apache InLong - Sort SDK ........................... SUCCESS [ 21.600 s]
[INFO] Apache InLong - DataProxy SDK ...................... SUCCESS [ 11.667 s]
[INFO] Apache InLong - TubeMQ Example ..................... SUCCESS [  7.208 s]
[INFO] Apache InLong - Audit .............................. SUCCESS [  3.327 s]
[INFO] Apache InLong - Audit Common ....................... SUCCESS [ 16.088 s]
[INFO] Apache InLong - Audit SDK .......................... SUCCESS [  7.024 s]
[INFO] Apache InLong - TubeMQ Server ...................... SUCCESS [ 57.408 s]
[INFO] Apache InLong - TubeMQ Connectors .................. SUCCESS [  0.817 s]
[INFO] Apache InLong - TubeMQ Connectors-flink ............ SUCCESS [  7.433 s]
[INFO] Apache InLong - TubeMQ Connectors-flume ............ SUCCESS [  5.358 s]
[INFO] Apache InLong - TubeMQ Connectors-spark ............ SUCCESS [ 10.977 s]
[INFO] Apache InLong - TubeMQ Manager ..................... SUCCESS [ 48.848 s]
[INFO] Apache InLong - TubeMQ Docker ...................... SUCCESS [  1.803 s]
[INFO] Apache InLong - TubeMQ Docker All .................. SUCCESS [  2.078 s]
[INFO] Apache InLong - TubeMQ Docker Build ................ SUCCESS [  1.170 s]
[INFO] Apache InLong - TubeMQ Docker Build C++ SDK Client . SUCCESS [  0.894 s]
[INFO] Apache InLong - TubeMQ Docker Manager .............. SUCCESS [  1.431 s]
[INFO] Apache InLong - Audit Proxy ........................ SUCCESS [ 16.066 s]
[INFO] Apache InLong - Audit store ........................ SUCCESS [ 19.609 s]
[INFO] Apache InLong - Audit Release ...................... SUCCESS [ 20.633 s]
[INFO] Apache InLong - Audit Docker ....................... SUCCESS [  4.083 s]
[INFO] Apache InLong - DataProxy .......................... SUCCESS [  3.326 s]
[INFO] Apache InLong - DataProxy Source ................... SUCCESS [ 23.141 s]
[INFO] Apache InLong - DataProxy Dist ..................... SUCCESS [  8.114 s]
[INFO] Apache InLong - DataProxy Docker ................... SUCCESS [  5.396 s]
[INFO] Apache InLong - Agent .............................. SUCCESS [  2.363 s]
[INFO] Apache InLong - Agent Common ....................... SUCCESS [ 23.694 s]
[INFO] Apache InLong - Agent Core ......................... SUCCESS [  8.050 s]
[INFO] Apache InLong - Agent Plugins ...................... SUCCESS [ 20.812 s]
[INFO] Apache InLong - Agent Release ...................... SUCCESS [ 11.910 s]
[INFO] Apache InLong - Agent Docker ....................... SUCCESS [  1.963 s]
[INFO] Apache InLong - Sort ............................... SUCCESS [  2.905 s]
[INFO] Apache InLong - Sort Formats ....................... SUCCESS [  2.794 s]
[INFO] Apache InLong - Sort Format-common ................. SUCCESS [ 16.259 s]
[INFO] Apache InLong - Sort Common ........................ SUCCESS [ 26.785 s]
[INFO] Apache InLong - Sort Api ........................... SUCCESS [  0.806 s]
[INFO] Apache InLong - Sort Format-base ................... SUCCESS [  4.793 s]
[INFO] Apache InLong - Sort Format-csv .................... SUCCESS [  4.419 s]
[INFO] Apache InLong - Sort Format-kv ..................... SUCCESS [  4.596 s]
[INFO] Apache InLong - Sort Flink ......................... SUCCESS [  2.730 s]
[INFO] Apache InLong - Sort Flink v1.13 ................... SUCCESS [  0.473 s]
[INFO] Apache InLong - Sort Flink Dependencies v1.13 ...... SUCCESS [  3.832 s]
[INFO] Apache InLong - Sort Format-inlongmsg-base ......... SUCCESS [ 15.158 s]
[INFO] Apache InLong - Sort Format-inlongmsg-csv .......... SUCCESS [  7.077 s]
[INFO] Apache InLong - Sort Format-json ................... SUCCESS [  8.086 s]
[INFO] Apache InLong - Sort Format-inlongmsg-pb ........... SUCCESS [  3.565 s]
[INFO] Apache InLong - Sort-connector-base ................ SUCCESS [ 13.661 s]
[INFO] Apache InLong - Sort Connectors v1.13 .............. SUCCESS [  1.526 s]
[INFO] Apache InLong - Sort-connector-hbase ............... SUCCESS [ 42.961 s]
[INFO] Apache InLong - Sort-connector-hive ................ SUCCESS [ 43.581 s]
[INFO] Apache InLong - Sort-connector-iceberg ............. SUCCESS [ 41.155 s]
[INFO] Apache InLong - Sort-connector-elasticsearch-base .. SUCCESS [ 13.633 s]
[INFO] Apache InLong - Sort-connector-elasticsearch6 ...... SUCCESS [ 35.072 s]
[INFO] Apache InLong - Sort-connector-elasticsearch7 ...... SUCCESS [ 35.804 s]
[INFO] Apache InLong - Sort-connector-jdbc ................ SUCCESS [ 24.421 s]
[INFO] Apache InLong - Sort-connector-redis ............... SUCCESS [ 19.645 s]
[INFO] Apache InLong - Sort-connector-kafka ............... SUCCESS [ 17.949 s]
[INFO] Apache InLong - Sort-connector-cdc-base ............ SUCCESS [  9.865 s]
[INFO] Apache InLong - Sort-connector-mongodb-cdc ......... SUCCESS [ 22.919 s]
[INFO] Apache InLong - Sort-connector-mysql-cdc ........... SUCCESS [ 19.779 s]
[INFO] Apache InLong - Sort-connector-postgres-cdc ........ SUCCESS [ 12.950 s]
[INFO] Apache InLong - Sort-connector-pulsar .............. SUCCESS [ 12.002 s]
[INFO] Apache InLong - Sort-connector-sqlserver-cdc ....... SUCCESS [ 14.633 s]
[INFO] Apache InLong - Sort-connector-oracle-cdc .......... SUCCESS [ 20.963 s]
[INFO] Apache InLong - Sort-connector-tubemq .............. SUCCESS [ 18.134 s]
[INFO] Apache InLong - Sort-connector-filesystem .......... SUCCESS [ 10.945 s]
[INFO] Apache InLong - Sort-connector-doris ............... SUCCESS [ 14.489 s]
[INFO] Apache InLong - Sort-connector-starrocks ........... SUCCESS [ 20.774 s]
[INFO] Apache InLong - Sort Core .......................... SUCCESS [ 17.860 s]
[INFO] Apache InLong - Sort Dist .......................... SUCCESS [  7.178 s]
[INFO] Apache InLong - Sort End to End Tests .............. SUCCESS [  2.189 s]
[INFO] Apache InLong - Sort-connector-hudi ................ SUCCESS [ 24.692 s]
[INFO] Apache InLong - Sort-connector-kudu ................ SUCCESS [ 19.301 s]
[INFO] Apache InLong - Sort Flink v1.15 ................... SUCCESS [  1.131 s]
[INFO] Apache InLong - Sort Flink Dependencies v1.15 ...... SUCCESS [  3.174 s]
[INFO] Apache InLong - Sort Connectors v1.15 .............. SUCCESS [  2.900 s]
[INFO] Apache InLong - Sort-connector-postgres-cdc ........ SUCCESS [ 24.793 s]
[INFO] Apache InLong - Sort Standalone .................... SUCCESS [ 13.387 s]
[INFO] Apache InLong - Sort Standalone Common ............. SUCCESS [ 15.649 s]
[INFO] Apache InLong - Sort Standalone Source ............. SUCCESS [ 17.096 s]
[INFO] Apache InLong - Sort Standalone Dist ............... SUCCESS [ 19.118 s]
[INFO] Apache InLong - Manager ............................ SUCCESS [  3.206 s]
[INFO] Apache InLong - Manager Test ....................... SUCCESS [ 10.453 s]
[INFO] Apache InLong - Manager Common ..................... SUCCESS [ 22.329 s]
[INFO] Apache InLong - Manager Pojo ....................... SUCCESS [ 56.976 s]
[INFO] Apache InLong - Manager Client ..................... SUCCESS [ 12.294 s]
[INFO] Apache InLong - Manager Client Examples ............ SUCCESS [  5.082 s]
[INFO] Apache InLong - Manager Client Tools ............... SUCCESS [  5.669 s]
[INFO] Apache InLong - Manager DAO ........................ SUCCESS [ 12.139 s]
[INFO] Apache InLong - Manager Workflow ................... SUCCESS [  4.658 s]
[INFO] Apache InLong - Manager Service .................... SUCCESS [ 15.520 s]
[INFO] Apache InLong - Manager Plugins .................... SUCCESS [  4.270 s]
[INFO] Apache InLong - Manager Web ........................ SUCCESS [ 19.789 s]
[INFO] Apache InLong - Manager Docker ..................... SUCCESS [  3.737 s]
[INFO] Apache InLong - Dashboard .......................... SUCCESS [01:36 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:34 min (Wall Clock)
[INFO] Finished at: 2023-06-14T19:40:05+08:00
[INFO] ------------------------------------------------------------------------

Full package:

mvn clean package

healchow
healchow reviewed Jun 14, 2023
View reviewed changes
pom.xml Show resolved Hide resolved
@healchow healchow changed the title [INLONG-8242][Tool] Separate the distribution module to improve the performance of CR pipeline [INLONG-8242][CI] Separate the distribution module to improve the performance of check pipeline Jun 14, 2023
@dockerzhang
Copy link
Contributor

@featzhang please update the workflow configurations, thanks.

dockerzhang
dockerzhang reviewed Jun 20, 2023
View reviewed changes
pom.xml
</modules>
</profile>
<profile>
<id>fast</id>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about fast change to parallel? Here we should use a more accurate word.

@healchow healchow self-requested a review June 21, 2023 08:46
@github-actions
Copy link

This PR is stale because it has been open for 60 days with no activity.

@github-actions github-actions bot added the stage/stale Issues or PRs that had no activity for a long time label Aug 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dockerzhang dockerzhang dockerzhang left review comments

@healchow healchow Awaiting requested review from healchow

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
stage/stale Issues or PRs that had no activity for a long time
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Improve][CI] Separate the distribution module to improve the performance of check pipeline
3 participants
@featzhang @dockerzhang @healchow