Use multi-stage builds in dockerfiles to reduce java images #9077

grubberr · 2021-12-23T08:23:33Z

Signed-off-by: Sergey Chvalyuk grubberr@gmail.com

What

We can reduce final docker image size by subtracting size of ${APPLICATION}.tar file
For example :

279M	build/distributions/destination-dynamodb-0.34.4-alpha.tar

How

One appoach we can do it using docker multi-stage builds:
on first stage we unpack and remove tar file and copy contants of /workdir on 2-nd stage

🚨 User Impact 🚨

No.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

This reverts commit b7ab21a.

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2021-12-23T11:06:27Z

/test connector=connectors/destination-dynamodb

🕑 connectors/destination-dynamodb https://github.com/airbytehq/airbyte/actions/runs/1615288451
❌ connectors/destination-dynamodb https://github.com/airbytehq/airbyte/actions/runs/1615288451
🐛

grubberr · 2021-12-23T11:34:47Z

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1615381681
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1615381681
No Python unittests run

grubberr · 2021-12-23T12:16:20Z

/test connector=connectors/destination-azure-blob-storage

🕑 connectors/destination-azure-blob-storage https://github.com/airbytehq/airbyte/actions/runs/1615507137
✅ connectors/destination-azure-blob-storage https://github.com/airbytehq/airbyte/actions/runs/1615507137
No Python unittests run

grubberr · 2021-12-23T13:13:13Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1615683834
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/1615683834
No Python unittests run

grubberr · 2021-12-23T13:13:57Z

/test connector=connectors/destination-csv

🕑 connectors/destination-csv https://github.com/airbytehq/airbyte/actions/runs/1615686355
✅ connectors/destination-csv https://github.com/airbytehq/airbyte/actions/runs/1615686355
No Python unittests run

grubberr · 2021-12-23T13:14:57Z

/test connector=connectors/destination-e2e-test

🕑 connectors/destination-e2e-test https://github.com/airbytehq/airbyte/actions/runs/1615689519
✅ connectors/destination-e2e-test https://github.com/airbytehq/airbyte/actions/runs/1615689519
No Python unittests run

grubberr · 2021-12-23T14:09:16Z

/test connector=connectors/source-mongodb-strict-encrypt

🕑 connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1615867330
❌ connectors/source-mongodb-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/1615867330
🐛 https://gradle.com/s/5xmxexivbsyaq

grubberr · 2021-12-23T14:09:32Z

/test connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1615868187
❌ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/1615868187
🐛 https://gradle.com/s/vajch457tpjge

grubberr · 2021-12-23T14:10:21Z

/test connector=connectors/source-snowflake

🕑 connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/1615871520

grubberr · 2021-12-23T14:10:51Z

/test connector=connectors/source-kafka

🕑 connectors/source-kafka https://github.com/airbytehq/airbyte/actions/runs/1615873216
✅ connectors/source-kafka https://github.com/airbytehq/airbyte/actions/runs/1615873216
No Python unittests run

alexandr-shegeda

looks great. we can proceed with merge without airbyte review

tuliren · 2021-12-31T20:06:40Z

@grubberr, it looks like that the master build started to fail on the Connectors Base step consistently after this PR was merged. Can you take a look and fix it?

grubberr · 2022-01-03T00:05:03Z

@tuliren there is a problem with class

./airbyte-integrations/bases/base-java/src/main/java/io/airbyte/integrations/destination/buffered_stream_consumer/BufferedStreamConsumer.java

and this tests

./airbyte-integrations/bases/base-java/src/test/java/io/airbyte/integrations/destination/buffered_stream_consumer/BufferedStreamConsumerTest.java

I have investigated a little and I see that this randomness RandomStringUtils.randomAscii

airbyte/airbyte-integrations/bases/base-java/src/test/java/io/airbyte/integrations/destination/buffered_stream_consumer/BufferedStreamConsumerTest.java

Line 318 in a36a860

    
           JsonNode payload = Jsons.jsonNode(ImmutableMap.of("id", RandomStringUtils.randomAscii(7), "name", "human " + String.format("%5d", i)));

can generate different size of final json objects
if total sum of all object = 1000 bytes -TESTS FAIL, if < 1000 - TESTS OK
we need to improve tests and I think improve BufferedStreamConsumer class itself

I will also investigate more about this problem

use multi-stage to reduce image size

b7ab21a

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

github-actions bot added the area/connectors Connector related issues label Dec 23, 2021

grubberr added 3 commits December 23, 2021 11:39

use multi-stage to reduce image size (2)

613040d

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Revert "use multi-stage to reduce image size"

723fd8d

This reverts commit b7ab21a.

use multi-stage to reduce image size (3)

a7500b9

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr requested a review from alexandr-shegeda December 23, 2021 09:46

grubberr temporarily deployed to more-secrets December 23, 2021 09:46 Inactive

grubberr linked an issue Dec 23, 2021 that may be closed by this pull request

Investigate Dockerfile changes to repair the integration tests #8996

Closed

2 tasks

jrhizor temporarily deployed to more-secrets December 23, 2021 11:08 Inactive

jrhizor temporarily deployed to more-secrets December 23, 2021 11:36 Inactive

jrhizor temporarily deployed to more-secrets December 23, 2021 12:18 Inactive

jrhizor temporarily deployed to more-secrets December 23, 2021 13:15 Inactive

jrhizor temporarily deployed to more-secrets December 23, 2021 13:16 Inactive

Merge branch 'master' into grubberr/8996-java-dockerfiles-multi-stage

c44ae27

jrhizor temporarily deployed to more-secrets December 23, 2021 14:11 Inactive

jrhizor temporarily deployed to more-secrets December 23, 2021 14:12 Inactive

alexandr-shegeda approved these changes Dec 23, 2021

View reviewed changes

Merge branch 'master' into grubberr/8996-java-dockerfiles-multi-stage

b803a97

grubberr merged commit 844dd93 into master Dec 23, 2021

grubberr deleted the grubberr/8996-java-dockerfiles-multi-stage branch December 23, 2021 20:27

grubberr temporarily deployed to more-secrets December 23, 2021 20:28 Inactive

jrhizor mentioned this pull request Dec 27, 2021

Bump Airbyte version from 0.35.1-alpha to 0.35.2-alpha #9119

Merged

karinakuz added connectors/destination/e2e-test connectors/destination/azure-blob-storage connectors/destinations-api labels Jan 5, 2022

karinakuz added connectors/destination/bigquery-denormalized connectors/destination/dynamodb connectors/destinations-files connectors/destinations-warehouse connectors/destinations-database and removed connectors/destinations-api labels Jan 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use multi-stage builds in dockerfiles to reduce java images #9077

Use multi-stage builds in dockerfiles to reduce java images #9077

grubberr commented Dec 23, 2021 •

edited

grubberr commented Dec 23, 2021 •

edited by github-actions bot

grubberr commented Dec 23, 2021 •

edited by github-actions bot

grubberr commented Dec 23, 2021 •

edited by github-actions bot

grubberr commented Dec 23, 2021 •

edited by github-actions bot

grubberr commented Dec 23, 2021 •

edited by github-actions bot

grubberr commented Dec 23, 2021 •

edited by github-actions bot

grubberr commented Dec 23, 2021 •

edited by github-actions bot

grubberr commented Dec 23, 2021 •

edited by github-actions bot

grubberr commented Dec 23, 2021 •

edited by github-actions bot

grubberr commented Dec 23, 2021 •

edited by github-actions bot

alexandr-shegeda left a comment

tuliren commented Dec 31, 2021 •

edited

grubberr commented Jan 3, 2022

Use multi-stage builds in dockerfiles to reduce java images #9077

Use multi-stage builds in dockerfiles to reduce java images #9077

Conversation

grubberr commented Dec 23, 2021 • edited

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

grubberr commented Dec 23, 2021 • edited by github-actions bot

grubberr commented Dec 23, 2021 • edited by github-actions bot

grubberr commented Dec 23, 2021 • edited by github-actions bot

grubberr commented Dec 23, 2021 • edited by github-actions bot

grubberr commented Dec 23, 2021 • edited by github-actions bot

grubberr commented Dec 23, 2021 • edited by github-actions bot

grubberr commented Dec 23, 2021 • edited by github-actions bot

grubberr commented Dec 23, 2021 • edited by github-actions bot

grubberr commented Dec 23, 2021 • edited by github-actions bot

grubberr commented Dec 23, 2021 • edited by github-actions bot

alexandr-shegeda left a comment

Choose a reason for hiding this comment

tuliren commented Dec 31, 2021 • edited

grubberr commented Jan 3, 2022

grubberr commented Dec 23, 2021 •

edited

grubberr commented Dec 23, 2021 •

edited by github-actions bot

grubberr commented Dec 23, 2021 •

edited by github-actions bot

grubberr commented Dec 23, 2021 •

edited by github-actions bot

grubberr commented Dec 23, 2021 •

edited by github-actions bot

grubberr commented Dec 23, 2021 •

edited by github-actions bot

grubberr commented Dec 23, 2021 •

edited by github-actions bot

grubberr commented Dec 23, 2021 •

edited by github-actions bot

grubberr commented Dec 23, 2021 •

edited by github-actions bot

grubberr commented Dec 23, 2021 •

edited by github-actions bot

grubberr commented Dec 23, 2021 •

edited by github-actions bot

tuliren commented Dec 31, 2021 •

edited