Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🎉 Destination Azure blob storage: introduced new connector with jsonl and csv formats #5332

Merged
merged 21 commits into from
Aug 29, 2021
Merged

🎉 Destination Azure blob storage: introduced new connector with jsonl and csv formats #5332

merged 21 commits into from
Aug 29, 2021

Conversation

etsybaev
Copy link
Contributor

@etsybaev etsybaev commented Aug 11, 2021 •
edited
Loading

What

A new Azure Blob Storage Destination introduced

How

This connector can write data in JSONL and CSV formats, override and append modes are supported.

Selection_305

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

  • Community member? Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • docs/SUMMARY.md
    • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
    • docs/integrations/README.md
    • airbyte-integrations/builds.md
  • PR name follows PR naming conventions
  • Connector added to connector index like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • Credentials added to Github CI. Instructions.
  • /test connector=connectors/<name> command is passing.
  • New Connector version released on Dockerhub by running the /publish command described here

ievgeniit added 13 commits July 17, 2021 23:30
[3447] Added skeleton for CHECK() and SPECK() methods
a5b25cb
Merge branch 'master' into etsybaev/3447-azure-blob-storage-destination
adc3e07
[3447] Added skeleton for JSONL writer (some of tests still fail, nee…
47c5fa2 
…d to fix)
Merge branch 'master' into etsybaev/3447-azure-blob-storage-destination
af87e82
[3447] Added some acceptance tests for azure blob storage jsonl client
8d8cc0d
[3447] minor refactoring
bf653d9
[3447] updated the way blob name is handled and check method's client
459fe72
[3447] Fixed check method if url was not set explicitly by customer
ab89ffd
[3447] Updated README.md
e82a3f8
[3447] Added basic CSV writer
20f527a
[3549] Destination:BigQuery: Added skeleton for AVRO (still doesn't w…
fe55528 
…or with APPEND option)
[3549] Destination:BigQuery: Added AVRO comment
eb4305e
[3549] Destination:BigQuery: cleared code
c927141
@etsybaev etsybaev self-assigned this Aug 11, 2021
@github-actions github-actions bot added the area/connectors Connector related issues label Aug 11, 2021
@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Aug 11, 2021
@etsybaev etsybaev changed the title [DRAFT_PR] Destination: Azure blob storage destination jsonl csv 🎉 Destination Azure blob storage: introduced new connector with jsonl and csv formats Aug 12, 2021
@jrhizor jrhizor temporarily deployed to more-secrets August 12, 2021 12:57 Inactive
@etsybaev
Copy link
Contributor Author

etsybaev commented Aug 12, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/destination-azure-blob-storage

🕑 connectors/destination-azure-blob-storage https://github.com/airbytehq/airbyte/actions/runs/1124901327
✅ connectors/destination-azure-blob-storage https://github.com/airbytehq/airbyte/actions/runs/1124901327

@jrhizor jrhizor temporarily deployed to more-secrets August 12, 2021 17:57 Inactive
@etsybaev
Copy link
Contributor Author

etsybaev commented Aug 12, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/destination-azure-blob-storage

🕑 connectors/destination-azure-blob-storage https://github.com/airbytehq/airbyte/actions/runs/1124956035
✅ connectors/destination-azure-blob-storage https://github.com/airbytehq/airbyte/actions/runs/1124956035

@jrhizor jrhizor temporarily deployed to more-secrets August 12, 2021 18:14 Inactive
@etsybaev etsybaev linked an issue Aug 16, 2021 that may be closed by this pull request
New Destination: Azure Blob Storage #3447
Closed
@irynakruk irynakruk self-requested a review August 17, 2021 09:15
irynakruk
irynakruk reviewed Aug 17, 2021
View reviewed changes
docs/SUMMARY.md Outdated Show resolved Hide resolved
@etsybaev etsybaev marked this pull request as ready for review August 17, 2021 14:36
@tuliren tuliren self-requested a review August 17, 2021 17:58
tuliren
tuliren suggested changes Aug 27, 2021
View reviewed changes
Copy link
Contributor

@tuliren tuliren left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fantastic!

Sorry about the delayed review.

We should not log every airbyte message. This the only thing that should be changed. All other comments are nitpicks.

...o/airbyte/integrations/destination/azure_blob_storage/AzureBlobStorageConnectionChecker.java Outdated Show resolved Hide resolved
...o/airbyte/integrations/destination/azure_blob_storage/AzureBlobStorageConnectionChecker.java Outdated Show resolved Hide resolved
...o/airbyte/integrations/destination/azure_blob_storage/AzureBlobStorageConnectionChecker.java Outdated Show resolved Hide resolved
...va/io/airbyte/integrations/destination/azure_blob_storage/csv/AzureBlobStorageCsvWriter.java Outdated Show resolved Hide resolved
...in/java/io/airbyte/integrations/destination/azure_blob_storage/AzureBlobStorageConsumer.java Outdated Show resolved Hide resolved
...o/airbyte/integrations/destination/azure_blob_storage/jsonl/AzureBlobStorageJsonlWriter.java Outdated Show resolved Hide resolved
.../airbyte/integrations/destination/azure_blob_storage/AzureBlobDestinationAcceptanceTest.java Outdated Show resolved Hide resolved
.../airbyte/integrations/destination/azure_blob_storage/AzureBlobDestinationAcceptanceTest.java Outdated Show resolved Hide resolved
docs/integrations/destinations/azureblobstorage.md Outdated Show resolved Hide resolved
docs/integrations/destinations/azureblobstorage.md Outdated Show resolved Hide resolved
@tuliren
Copy link
Contributor

tuliren commented Aug 27, 2021

Also don't forget to add this destination to the connector catalog in docs/integrations/README.md.

@etsybaev
Copy link
Contributor Author

etsybaev commented Aug 28, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/destination-azure-blob-storage

🕑 connectors/destination-azure-blob-storage https://github.com/airbytehq/airbyte/actions/runs/1177520311
✅ connectors/destination-azure-blob-storage https://github.com/airbytehq/airbyte/actions/runs/1177520311

@jrhizor jrhizor temporarily deployed to more-secrets August 28, 2021 16:36 Inactive
@etsybaev
Copy link
Contributor Author

to the connector catalog in docs/integrations/README.md.

It's already there. Thanks

tuliren
tuliren approved these changes Aug 28, 2021
View reviewed changes
Copy link
Contributor

@tuliren tuliren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! Thank you~~

@etsybaev
Copy link
Contributor Author

etsybaev commented Aug 28, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/destination-azure-blob-storage

🕑 connectors/destination-azure-blob-storage https://github.com/airbytehq/airbyte/actions/runs/1177939776
✅ connectors/destination-azure-blob-storage https://github.com/airbytehq/airbyte/actions/runs/1177939776

@jrhizor jrhizor temporarily deployed to more-secrets August 28, 2021 21:18 Inactive
@etsybaev
Copy link
Contributor Author

etsybaev commented Aug 29, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/destination-azure-blob-storage

🕑 connectors/destination-azure-blob-storage https://github.com/airbytehq/airbyte/actions/runs/1178856541
✅ connectors/destination-azure-blob-storage https://github.com/airbytehq/airbyte/actions/runs/1178856541

@jrhizor jrhizor temporarily deployed to more-secrets August 29, 2021 07:49 Inactive
@etsybaev
Copy link
Contributor Author

etsybaev commented Aug 29, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-azure-blob-storage

🕑 connectors/destination-azure-blob-storage https://github.com/airbytehq/airbyte/actions/runs/1178896845
✅ connectors/destination-azure-blob-storage https://github.com/airbytehq/airbyte/actions/runs/1178896845

@jrhizor jrhizor temporarily deployed to more-secrets August 29, 2021 08:17 Inactive
@etsybaev etsybaev merged commit 61842ed into master Aug 29, 2021
@etsybaev etsybaev deleted the etsybaev/3447-azure-blob-storage-destination-jsonl-csv branch August 29, 2021 09:24
@jrhizor jrhizor mentioned this pull request Sep 3, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tuliren tuliren tuliren approved these changes

@irynakruk irynakruk irynakruk approved these changes

@yaroslav-hrytsaienko yaroslav-hrytsaienko Awaiting requested review from yaroslav-hrytsaienko

Assignees

@etsybaev etsybaev

Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/destination/azure-blob-storage connectors/destinations-files
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

New Destination: Azure Blob Storage
5 participants
@etsybaev @tuliren @irynakruk @jrhizor @karinakuz