-
Notifications
You must be signed in to change notification settings - Fork 2.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(spark-test): upgrade gradle and fix spark smoke test #8777
fix(spark-test): upgrade gradle and fix spark smoke test #8777
Conversation
…ahub into spark-smoke-test-fix-1
0001660
to
fe97ad3
Compare
018b0ac
to
9225f6b
Compare
f3b6828
to
bade513
Compare
WORKDIR /datahub-ingestion | ||
RUN sed -i.bak "s/__version__ = \"0.0.0.dev0\"/__version__ = \"$RELEASE_VERSION\"/" src/datahub/__init__.py && \ | ||
cat src/datahub/__init__.py && \ | ||
chown -R datahub /datahub-ingestion |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was really hoping to avoid having lots of dockerfiles - is it possible to do this stuff using build args?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lack of support for target in the gradle docker plugin, if i merge then it'll build more then it needs and takes longer for the spark smoke test
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is for a specific instance, I don't think it's indicative of a proliferation of Dockerfiles in general, so I think it's probably okay. If we start needing slim images of others as well we can think about getting tricky with build args, profiles, etc. is my two cents 😄
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These changes LGTM, but I don't think it actually solves the issue of the images not building
Seems like the rules need to be tweaked
if: ${{ steps.filter.outputs.datahub-ingestion-base == 'true' || steps.filter.outputs.datahub-ingestion == 'true' }}
should also have a check of <is release / commit to master> || (... original condition)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let's see if this does the trick
* tag 'v0.11.0': (188 commits) fix(spark-test): upgrade gradle and fix spark smoke test (datahub-project#8777) fix(gms): Fixed Recently Viewed section for users with '@' in the URN. (datahub-project#8754) feat: add feedback widget (datahub-project#8732) fix(custom-search): fix custom search to be able to use unquoted query (datahub-project#8805) docs(db-retention): update with default setting (datahub-project#8797) feat(openapi): entity endpoints & analytics raw (datahub-project#8537) feat(search): Also de-duplicate the field queries based on field names (datahub-project#8788) fix(ingest): drop `wrap_aspect_as_workunit` method (datahub-project#8766) feat(ingest): drop sql_metadata parser (datahub-project#8765) docs: minor fix on versioning navbar and dropdown (datahub-project#8790) chore(ingest): upgrade sqlglot fork (datahub-project#8775) docs: add datahub source to integrations page (datahub-project#8787) fix(ingest/bigquery): fix partition and median queries for profiling (datahub-project#8778) fix(ingest/tableau): fix tableau native CLL for snowflake, add type annotations (datahub-project#8779) refactor(ingest): Add support for group-owners in dataflow entities (datahub-project#8154) feat(systemMetadata): Adding a lastRunId field system metadata (datahub-project#8672) feat(airflow-plugin): add package type information (datahub-project#8795) fix(ingest/datahub): Support postgres; build(postgres): Modernize postgres docker setup (datahub-project#8762) docs(session): add documentation for session token duration and fix default (datahub-project#8791) chore(analytics): bump version (datahub-project#8786) ...
Checklist