chore(ci): fix flakiness, misc improvements #7605
Conversation
github-actions
bot
added
the
devops
anshbansal
changed the title
anshbansal
added
ingestion
| id: cache-venv | ||
| with: | ||
| path: ./smoke-test/venv/ | ||
| key: smoke-venv-${{ runner.os }}-${{ steps.packages_checksum.outputs.packages_checksum }} |
There was a problem hiding this comment.
Can we comment this out instead? Do you actually know what the issue was here or just disabling because of issues?
There was a problem hiding this comment.
There is no difference in run time. See master and this branch's runs. No time difference is there.
There was a problem hiding this comment.
Weird - that probably means the caching wasn't even working?
| path: ./metadata-ingestion/venv/ | ||
| key: ${{ runner.os }}-venv-${{ steps.packages_checksum.outputs.packages_checksum }}-${{ matrix.python-version }} | ||
| - name: Install package | ||
| run: ./gradlew :metadata-ingestion:installPackageOnly |
There was a problem hiding this comment.
Is this just for debugging?
There was a problem hiding this comment.
No. Wanted to see how much time installing the package takes separately to the tests.
| @@ -12,7 +12,8 @@ if (!project.hasProperty("extra_pip_requirements")) { | |||
| } | |||
|
|
|||
| task checkPythonVersion(type: Exec) { | |||
| commandLine python_executable, '-c', 'import sys; assert sys.version_info >= (3, 7)' | |||
| commandLine python_executable, '-c', | |||
| 'import sys; assert (3, 11) > sys.version_info >= (3, 7), f"Python version {sys.version_info[:2]} not allowed"' | |||
There was a problem hiding this comment.
Change message to "Is too old"
There was a problem hiding this comment.
What do you mean? If someone is using 3.11 it won't be too old but too new.
There was a problem hiding this comment.
oh i missed the 3.11 part - then this is fine
|
Also it seems strange that we've seen issues in smoke test but not with metadata ingestion yet. Any ideas why that could be? |
venvcontainspython. Caching wholevenvinstead of justpipcache means you are backing up python which is problematic and is probably causing python not being found randomly in the smoke tests. If it is causing issues with smoke tests it would also cause issues withmetadata-ingestionso removing that too.e.g. https://github.com/datahub-project/datahub/actions/runs/4433850334/jobs/7779664856 where log says
Python not being found is really odd as we have a setup python step. It is probably the
venvcache after that which is messing things somehow.Checklist