airbyte-ci: refactor gradle containers, use amazoncorretto #30384

postamar · 2023-09-13T15:08:09Z

This PR collects a bunch of airbyte-ci improvements which were discovered by @alafanechere and myself while on a bug hunt. These improvements are centred on the GradleTask.

postamar · 2023-09-13T15:16:11Z

Testing this change on:

source-postgres: https://github.com/airbytehq/airbyte/actions/runs/6174412610/job/16758953944
destination-bigquery: https://github.com/airbytehq/airbyte/actions/runs/6175169779/job/16761436561
destination-snowflake: https://github.com/airbytehq/airbyte/actions/runs/6175187070/job/16761839399

alafanechere

Just a partial review as I have to switch context. Will come back at it soon!

airbyte-ci/connectors/pipelines/pipelines/actions/environments.py

airbyte-ci/connectors/pipelines/pipelines/consts.py

airbyte-ci/connectors/pipelines/pipelines/contexts.py

alafanechere · 2023-09-13T16:59:59Z

airbyte-ci/connectors/pipelines/pipelines/actions/environments.py

+        # Mount /var/lib/docker so docker writes to the host file system, i.e. the dagger engine.
+        .with_mounted_cache("/var/lib/docker", dagger_client.cache_volume("shared-var-lib-docker"))


The thing I noticed in the past with this kind of caching is that when Dagger kills the service dockerd is not always gracefully exited, it means it can corrupt the /var/lib/docker content and prevent a fine reboot of the service. But it's worth a new try.

In this case all we want is this to be a volume, to avoid storage engine issues in dind. We don't actually care about the caching functionality. Perhaps we can add an rm -rf /var/lib/docker in the with_exec below to nuke the cache?

avoid storage engine issues in dind

These were the reason of other errors/warning logs from dockerd right?

Yes. I think I witnessed instances of this corruption. I'm going to try using a different storage engine, fuse-overlayfs. I have no idea what I'm doing.

alafanechere

Super cool readbility boost. Minor suggestion, remarks and improvement question. A lot of these suggestions are basically suggestion about changing things I did in the past that you did not introduce :D
And please: bump the airbyte-ci version in pyproject.toml (we should definitely add a git hook for this).

airbyte-ci/connectors/pipelines/pipelines/actions/environments.py

airbyte-ci/connectors/pipelines/pipelines/builds/java_connectors.py

airbyte-ci/connectors/pipelines/pipelines/gradle.py

alafanechere · 2023-09-13T19:37:37Z

Testing this change on:

source-postgres: https://github.com/airbytehq/airbyte/actions/runs/6174412610/job/16758953944
destination-bigquery: https://github.com/airbytehq/airbyte/actions/runs/6175169779/job/16761436561
destination-snowflake: https://github.com/airbytehq/airbyte/actions/runs/6175187070/job/16761839399

All the test passed 👍 . Just a reminder: the workflow successful execution is not correlated with the test success: you can have correct pipeline execution with failing tests. I checked the test report in the workflow logs. Otherwise a trick is to push an harmless change on a connector file (that you revert after): it will trigger airbyte-ci on the branch and report test statuses.

postamar · 2023-09-13T20:29:54Z

Thanks for the review!

And please: bump the airbyte-ci version in pyproject.toml (we should definitely add a git hook for this).

I'm sorry I keep forgetting this 😬
Instead of a git hook, perhaps a required check in github? Non-empty diff in airbyte-ci/ should require a different version than in master. It can be done.

you can have correct pipeline execution with failing tests.

I noticed! I think I'll do just that.

github-actions · 2023-09-13T21:02:09Z

destination-snowflake test report (commit `0efd66e578`) - ❌

⏲️ Total pipeline duration: 10mn47s

Step	Result
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

github-actions · 2023-09-13T21:09:02Z

destination-bigquery test report (commit `0efd66e578`) - ❌

⏲️ Total pipeline duration: 06mn43s

Step	Result
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

github-actions · 2023-09-13T21:10:01Z

Coverage report for source-postgres

File	Coverage [32.18%]	❌
PostgresSourceOperations.java	32.18%	❌

Total Project Coverage	70.6%	🍏

github-actions · 2023-09-13T21:33:02Z

source-postgres test report (commit `0efd66e578`) - ❌

⏲️ Total pipeline duration: 23mn49s

Step	Result
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

postamar · 2023-09-13T21:43:48Z

/approve-and-merge reason="not actually a connector change"

postamar · 2023-09-13T21:44:10Z

Thanks for the review! All is well. Merging.

octavia-approvington · 2023-09-13T21:44:26Z

This is really good

github-actions · 2023-09-13T21:49:19Z

destination-snowflake test report (commit `c32efc5d33`) - ❌

⏲️ Total pipeline duration: 04mn05s

Step	Result
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

github-actions · 2023-09-13T21:52:22Z

destination-bigquery test report (commit `c32efc5d33`) - ❌

⏲️ Total pipeline duration: 02mn50s

Step	Result
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

github-actions · 2023-09-13T22:18:21Z

source-postgres test report (commit `c32efc5d33`) - ❌

⏲️ Total pipeline duration: 25mn48s

Step	Result
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Acceptance tests	✅
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

airbyte-ci: refactor gradle containers, use amazoncorretto

57cf56c

postamar requested review from a team and aaronsteers September 13, 2023 16:47

postamar mentioned this pull request Sep 13, 2023

bump testcontainers version to latest #30392

Closed

alafanechere reviewed Sep 13, 2023

View reviewed changes

postamar added 3 commits September 13, 2023 14:01

apply review comments

5a19a75

fixup previous commit

4c04068

unit tests require docker

e7b1122

alafanechere approved these changes Sep 13, 2023

View reviewed changes

postamar added 6 commits September 13, 2023 15:46

remove 10 second sleep

e0cd84c

use fuse-overlayfs

fe5e2c8

use fuse-overlayfs

2674377

add element to DEFAULT_EXCLUDED_FILES list

cdb80c6

bump version and update changelog

56997dc

docker magic constants

b198198

postamar added 4 commits September 13, 2023 16:30

misc small fixes

86adb33

misc small changes

3ab880e

apply gradle format to airbyte-ci

148e99f

touch 3 connectors to trigger airbyte-ci

0efd66e

postamar requested review from a team as code owners September 13, 2023 20:49

octavia-squidington-iii added area/connectors Connector related issues connectors/destination/bigquery connectors/destination/snowflake connectors/source/postgres labels Sep 13, 2023

Merge branch 'master' into postamar/airbyte-ci-uses-amazoncorretto

c32efc5

octavia-approvington approved these changes Sep 13, 2023

View reviewed changes

octavia-approvington merged commit 25edee4 into master Sep 13, 2023
9 of 10 checks passed

octavia-approvington deleted the postamar/airbyte-ci-uses-amazoncorretto branch September 13, 2023 21:44

postamar mentioned this pull request Sep 14, 2023

Gradle not cached, installed on every run #29111

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

airbyte-ci: refactor gradle containers, use amazoncorretto #30384

airbyte-ci: refactor gradle containers, use amazoncorretto #30384

postamar commented Sep 13, 2023

postamar commented Sep 13, 2023 •

edited

Loading

alafanechere left a comment

alafanechere Sep 13, 2023

postamar Sep 13, 2023

alafanechere Sep 13, 2023 •

edited

Loading

postamar Sep 13, 2023

alafanechere left a comment •

edited

Loading

alafanechere commented Sep 13, 2023

postamar commented Sep 13, 2023

github-actions bot commented Sep 13, 2023

github-actions bot commented Sep 13, 2023

github-actions bot commented Sep 13, 2023 •

edited

Loading

github-actions bot commented Sep 13, 2023

postamar commented Sep 13, 2023

postamar commented Sep 13, 2023

octavia-approvington commented Sep 13, 2023

github-actions bot commented Sep 13, 2023

github-actions bot commented Sep 13, 2023

github-actions bot commented Sep 13, 2023

		# Mount /var/lib/docker so docker writes to the host file system, i.e. the dagger engine.
		.with_mounted_cache("/var/lib/docker", dagger_client.cache_volume("shared-var-lib-docker"))

airbyte-ci: refactor gradle containers, use amazoncorretto #30384

airbyte-ci: refactor gradle containers, use amazoncorretto #30384

Conversation

postamar commented Sep 13, 2023

postamar commented Sep 13, 2023 • edited Loading

alafanechere left a comment

Choose a reason for hiding this comment

alafanechere Sep 13, 2023

Choose a reason for hiding this comment

postamar Sep 13, 2023

Choose a reason for hiding this comment

alafanechere Sep 13, 2023 • edited Loading

Choose a reason for hiding this comment

postamar Sep 13, 2023

Choose a reason for hiding this comment

alafanechere left a comment • edited Loading

Choose a reason for hiding this comment

alafanechere commented Sep 13, 2023

postamar commented Sep 13, 2023

github-actions bot commented Sep 13, 2023

destination-snowflake test report (commit 0efd66e578) - ❌

github-actions bot commented Sep 13, 2023

destination-bigquery test report (commit 0efd66e578) - ❌

github-actions bot commented Sep 13, 2023 • edited Loading

Coverage report for source-postgres

github-actions bot commented Sep 13, 2023

source-postgres test report (commit 0efd66e578) - ❌

postamar commented Sep 13, 2023

postamar commented Sep 13, 2023

octavia-approvington commented Sep 13, 2023

github-actions bot commented Sep 13, 2023

destination-snowflake test report (commit c32efc5d33) - ❌

github-actions bot commented Sep 13, 2023

destination-bigquery test report (commit c32efc5d33) - ❌

github-actions bot commented Sep 13, 2023

source-postgres test report (commit c32efc5d33) - ❌

postamar commented Sep 13, 2023 •

edited

Loading

alafanechere Sep 13, 2023 •

edited

Loading

alafanechere left a comment •

edited

Loading

destination-snowflake test report (commit `0efd66e578`) - ❌

destination-bigquery test report (commit `0efd66e578`) - ❌

github-actions bot commented Sep 13, 2023 •

edited

Loading

source-postgres test report (commit `0efd66e578`) - ❌

destination-snowflake test report (commit `c32efc5d33`) - ❌

destination-bigquery test report (commit `c32efc5d33`) - ❌

source-postgres test report (commit `c32efc5d33`) - ❌