Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🎉 New Destination: Starburst Galaxy #24620

Closed
wants to merge 1 commit into from
Closed

🎉 New Destination: Starburst Galaxy #24620

wants to merge 1 commit into from

Conversation

mayankvadariya
Copy link
Contributor

@mayankvadariya mayankvadariya commented Mar 27, 2023 •
edited

What

Add Starburst Galaxy destination connector. This connector will unlock new capabilities for the customers having data lake analytics usecases. 🎉🎉🎉

How

  • Creates temp Iceberg table using Iceberg's Java APIs
  • Register the creates temp table through Galaxy SQL statement
  • Merge the temp table data to target table

It uses JDBC drivers to connect to Starburst Galaxy cluster.

To run the integration test, create secrets/config.json in the connector root folder. Sample config file is located at sample_secrets/config.json under connector root folder.

Recommended reading order

  1. starburst-galaxy.md
  2. StarburstGalaxyDestination.java, StarburstGalaxyS3StreamCopier.java

Pre-merge Checklist

New Connector

Community member or Airbyter

  • Community member? Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Connector version is set to 0.0.1
    • Dockerfile has version 0.0.1
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • docs/integrations/<source or destination>/<name>.md including changelog with an entry for the initial version. See changelog example
    • docs/integrations/README.md
    • airbyte-integrations/builds.md
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • If new credentials are required for use in CI, add them to GSM. Instructions.
  • /test connector=connectors/<name> command is passing
  • New Connector version released on Dockerhub by running the /publish command described here
  • After the connector is published, connector added to connector index as described here
  • Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

@mayankvadariya mayankvadariya requested a review from a team as a code owner March 27, 2023 22:52
@CLAassistant
Copy link

CLAassistant commented Mar 27, 2023 •
edited

CLA assistant check
All committers have signed the CLA.

@mayankvadariya mayankvadariya force-pushed the add-galaxy-dest-connector branch 3 times, most recently from 2075962 to dcea7de Compare March 31, 2023 02:57
evantahler
evantahler reviewed Apr 1, 2023
View reviewed changes
Copy link
Contributor

@evantahler evantahler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you please put our Github Actions back :D

@mayankvadariya
Copy link
Contributor Author

can you please put our Github Actions back :D

Yes, removed that commit :)

marcosmarxm
marcosmarxm requested changes Apr 2, 2023
View reviewed changes
Copy link
Member

@marcosmarxm marcosmarxm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick review. Tomorrow I'll take a deeper look.

...s/base-java-s3/src/main/java/io/airbyte/integrations/destination/s3/avro/JsonSchemaType.java Outdated Show resolved Hide resolved
...-s3/src/main/java/io/airbyte/integrations/destination/s3/avro/JsonToAvroSchemaConverter.java Outdated Show resolved Hide resolved
...te-integrations/bases/standard-destination-test/src/main/resources/v0/edge_case_catalog.json Outdated Show resolved Hide resolved
@mayankvadariya mayankvadariya force-pushed the add-galaxy-dest-connector branch 2 times, most recently from 3234ca5 to 177f03f Compare April 4, 2023 16:39
@marcosmarxm
Copy link
Member

marcosmarxm commented Apr 19, 2023 •
edited by github-actions bot

/test connector=connectors/destination-starburst-galaxy

🕑 connectors/destination-starburst-galaxy https://github.com/airbytehq/airbyte/actions/runs/4746051650
❌ connectors/destination-starburst-galaxy https://github.com/airbytehq/airbyte/actions/runs/4746051650
🐛 https://gradle.com/s/5yykpp2p4xrte

Build Failed

Test summary info:

Could not find result summary

@marcosmarxm
Copy link
Member

marcosmarxm commented Apr 19, 2023 •
edited by github-actions bot

/test connector=connectors/destination-starburst-galaxy

🕑 connectors/destination-starburst-galaxy https://github.com/airbytehq/airbyte/actions/runs/4746510636
✅ connectors/destination-starburst-galaxy https://github.com/airbytehq/airbyte/actions/runs/4746510636
No Python unittests run

Build Passed

Test summary info:

All Passed

@marcosmarxm
Copy link
Member

marcosmarxm commented Apr 20, 2023 •
edited by github-actions bot

/test connector=connectors/destination-starburst-galaxy

🕑 connectors/destination-starburst-galaxy https://github.com/airbytehq/airbyte/actions/runs/4755258814
✅ connectors/destination-starburst-galaxy https://github.com/airbytehq/airbyte/actions/runs/4755258814
No Python unittests run

Build Passed

Test summary info:

All Passed

@marcosmarxm
Copy link
Member

marcosmarxm commented Apr 20, 2023 •
edited by github-actions bot

/test connector=connectors/destination-starburst-galaxy

🕑 connectors/destination-starburst-galaxy https://github.com/airbytehq/airbyte/actions/runs/4757580225

@marcosmarxm
Copy link
Member

marcosmarxm commented Apr 20, 2023 •
edited by github-actions bot

/test connector=connectors/destination-starburst-galaxy

🕑 connectors/destination-starburst-galaxy https://github.com/airbytehq/airbyte/actions/runs/4758093726
✅ connectors/destination-starburst-galaxy https://github.com/airbytehq/airbyte/actions/runs/4758093726
No Python unittests run

Build Passed

Test summary info:

All Passed

@sh4sh sh4sh mentioned this pull request Apr 21, 2023
Merged
42 tasks
@marcosmarxm
Copy link
Member

Thanks for the contribution @mayankvadariya the code was merged at #25399

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@evantahler evantahler evantahler left review comments

@grantatspothero grantatspothero grantatspothero left review comments

@marcosmarxm marcosmarxm marcosmarxm approved these changes

Assignees
No one assigned
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation bounty community connectors/destination/starburst-galaxy new-destination
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@mayankvadariya @CLAassistant @marcosmarxm @evantahler @grantatspothero @octavia-squidington-iii