Repository navigation
🐛 Destination S3: Fix Parquet LZO compression - #26284
Conversation
This reverts commit 1241569.
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
|
/test connector=connectors/destination-s3
Build PassedTest summary info: |
Affected Connector ReportNOTE
|
| Connector | Version | Changelog | Publish |
|---|
- See "Actionable Items" below for how to resolve warnings and errors.
❌ Destinations (10)
| Connector | Version | Changelog | Publish |
|---|---|---|---|
destination-bigquery |
1.4.3 |
✅ | ✅ |
destination-bigquery-denormalized |
1.4.1 |
✅ | ✅ |
destination-databricks |
1.0.2 |
✅ | ✅ |
destination-gcs |
0.3.0 |
✅ | ✅ |
destination-r2 |
0.1.0 |
✅ | ✅ |
destination-redshift |
0.4.7 |
✅ | ✅ |
destination-s3 |
0.4.1 |
✅ | ❌ (diff seed version) |
destination-s3-glue |
0.1.7 |
✅ | ✅ |
destination-snowflake |
1.0.4 |
✅ | ✅ |
destination-starburst-galaxy |
0.0.1 |
✅ | ✅ |
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Other Modules (0)
Actionable Items
(click to expand)
| Category | Status | Actionable Item |
|---|---|---|
| Version | ❌ mismatch |
The version of the connector is different from its normal variant. Please bump the version of the connector. |
| ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
|
| Changelog | ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
| ❌ changelog missing |
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog. | |
| Publish | ⚠ not in seed |
The connector is not in the cloud or oss registry, so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that you have added a metadata.yaml file and the expected registries are enabled. |
|
hm, test is passing but I'm still seeing errors when I try running a sync in oss :/ |
|
nope, I'm just dumb and built the image from the wrong branch 🤦 @airbytehq/destinations ptal :D |
|
/test connector=connectors/destination-gcs
Build PassedTest summary info: |
| * Only verifies that it runs successfully, which is sufficient to catch any issues with installing the lzo libraries. | ||
| */ | ||
| @Test | ||
| public void testLzoCompression() throws Exception { |
|
/test connector=connectors/destination-s3
Build PassedTest summary info: |
|
merging without publish because we have publish on merge! |
|
but there's (presumably?) a diff between the dockerfile and how dagger is configured :( lzo sync still failing on cloud https://cloud.airbyte.com/workspaces/46b4844a-94a3-496f-af53-2050bf4141af/connections/79b4d19f-2077-461a-9d4b-cfaa6c9538dc/job-history#2292715::0 |
* Revert "Move hadoop-lzo to test dependency (#21085)" This reverts commit 1241569. * add basic test * Automated Change * version bumps, changelog * Automated Change * unused import * Ran ./gradlew :spotlessJavaApply to trigger GitHub build * regenerate registry --------- Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: ryankfu <ryan.fu@airbyte.io>
* Revert "Move hadoop-lzo to test dependency (airbytehq#21085)" This reverts commit 1241569. * add basic test * Automated Change * version bumps, changelog * Automated Change * unused import * Ran ./gradlew :spotlessJavaApply to trigger GitHub build * regenerate registry --------- Co-authored-by: edgao <edgao@users.noreply.github.com> Co-authored-by: ryankfu <ryan.fu@airbyte.io>

revert #21085 + add a test case to actually run an lzo compression sync. seen in https://github.com/airbytehq/oncall/issues/2028
the test fails without the revert + passes after the revert.
it's still broken in destination-gcs, but I don't want to install java 8 + lzo libraries there... issue to do something about this https://github.com/airbytehq/airbyte-internal-issues/issues/1774