[Chore] Improve the robustness of CI #16592
Conversation
| if [ ! -f /tmp/${{ matrix.version }}/apache-dolphinscheduler-${{ matrix.version }}-bin.tar.gz ]; then | ||
| echo "Binary package not found in cache, downloading..." | ||
| mkdir -p /tmp/${{ matrix.version }} | ||
| wget https://mirrors.tuna.tsinghua.edu.cn/apache/dolphinscheduler/${{ matrix.version }}/apache-dolphinscheduler-${{ matrix.version }}-bin.tar.gz -P /tmp/${{ matrix.version }} |
There was a problem hiding this comment.
Since https://archive.apache.org download speed is too slow and often causes timeout failures. So using https://mirrors.tuna.tsinghua.edu.cn to do the cache instead.
Notice: This site does not contains the archived versions.
There was a problem hiding this comment.
not sure use tsinghua mirrors is good idea or not
There was a problem hiding this comment.
if accept can we add some desc in L144?
There was a problem hiding this comment.
not sure use tsinghua mirrors is good idea or not
After I tested a dozen mirrors, I found this source to be the fastest in github action.
if accept can we add some desc in L144?
It seems comments don't seem to be supported in run. I tried, but I got an error...
There was a problem hiding this comment.
how about using apache cdn, such as https://dlcdn.apache.org/dolphinscheduler/3.2.2/apache-dolphinscheduler-3.2.2-bin.tar.gz for version 3.2.2?
There was a problem hiding this comment.
It seems comments don't seem to be supported in run. I tried, but I got an error...
I mean we and comment before the line for run property.
There was a problem hiding this comment.
how about using apache cdn, such as
https://dlcdn.apache.org/dolphinscheduler/3.2.2/apache-dolphinscheduler-3.2.2-bin.tar.gzfor version 3.2.2?
And how about apache official cdn?
There was a problem hiding this comment.
how about using apache cdn, such as
https://dlcdn.apache.org/dolphinscheduler/3.2.2/apache-dolphinscheduler-3.2.2-bin.tar.gzfor version 3.2.2?
This is the same speed as archive.apache.org...
There was a problem hiding this comment.
all right, so maybe we can keep tsinghua mirror here
SbloodyS
requested review from
Gallardot,
zhongjiajie,
ruanwenjun,
EricGao888 and
Radeity
|
| @@ -151,7 +151,7 @@ jobs: | |||
| uses: actions/cache@v4 | |||
| with: | |||
| path: ~/.m2/repository | |||
| key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}-e2e | |||
| key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}-backend | |||
There was a problem hiding this comment.
e2e has different pom.xml with the backend module, as you can see it uses dolphinscheduler-e2e/pom.xml instead of /pom.xml
There was a problem hiding this comment.
It doesn't matter. The maven cache path is ~/.m2/repository. The old dependency cache is retained and new dependencies are added.
There was a problem hiding this comment.
I have half a memory we have some error with the same cache key #14808 but I am not sure, maybe we can keep this change to see whether regression in the further
There was a problem hiding this comment.
Thanks for clarification.
| @@ -62,7 +62,7 @@ jobs: | |||
| uses: actions/cache@v4 | |||
| with: | |||
| path: ~/.m2/repository | |||
| key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}-api-test | |||
| key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}-backend | |||
There was a problem hiding this comment.
The total cache storage limit is 10GB. We add binary package to it will cause overuse. So I reuse it. Please refer to the doc in
https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy
Purpose of the pull request
Brief change log
Verify this pull request
This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(or)
Pull Request Notice
Pull Request Notice
If your pull request contain incompatible change, you should also add it to
docs/docs/en/guide/upgrede/incompatible.md