Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AirbyteLib example notebook with source-faker #33878

Closed

Conversation

aaronsteers
Copy link
Collaborator

@aaronsteers aaronsteers commented Jan 3, 2024

This adds a working example notebook and source-faker as a source. Also fixes bugs discovered during execution of the sample source.

Should merge after:

Joe Reuter and others added 30 commits December 13, 2023 10:32
improve sql handling, add duckdb and pgsql

poetry init

cleanup imports

add pandas option

delete stale file

set parquet config class

Merge 'origin/flash1293/airbyte-lib' into aj/airbyt-lib-caches

update deps

move all non-public methods to be protected members

consolidate public interface in code file

delete files from missing .gitignore rules

decouple file writers from caches

mypy fixes, refactor filewriters,

cleanup finalize_batches implementation
improve sql handling, add duckdb and pgsql

poetry init

cleanup imports

add pandas option

delete stale file

set parquet config class

Merge 'origin/flash1293/airbyte-lib' into aj/airbyt-lib-caches

update deps

move all non-public methods to be protected members

consolidate public interface in code file

delete files from missing .gitignore rules

decouple file writers from caches

mypy fixes, refactor filewriters,

cleanup finalize_batches implementation

cleanup sql cache implementation
@stephane-airbyte
Copy link
Contributor

@aaronsteers looks like you had a bad merge here. You'll need to rebase

@octavia-squidington-iii octavia-squidington-iii removed CDK Connector Development Kit area/documentation Improvements or additions to documentation labels Jan 8, 2024
@aaronsteers
Copy link
Collaborator Author

aaronsteers commented Jan 8, 2024

@flash1293 and @alafanechere Thank you both for your feedback here on this PR. My plan is to split this into a few different places:

  1. Move pip_url support into a new PR against `master. (Done.)
  2. Move other fixes related to caches into the PR that this was based on:
  3. Move source-faker update to a dedicated PR. (Done.)
  4. Keep example notebook in its own PR, or merge with the devcontainer PR (if ready).

I'll note also that I don't have full context on the action items in this comment regarding ConnectorMetadataDefinitionsV0 but it sounds like the next step here will be available when CDK is published to PyPi.

Update:

@alafanechere
Copy link
Contributor

I'll note also that I don't have full context on the action items in #33878 (comment) regarding ConnectorMetadataDefinitionsV0 but it sounds like the next step here will be available when CDK is published to PyPi.

We discussed with @flash1293 that this will be addressed once the metadata lib package will be published to PyPi (it's not in the CDK). This will be enabled by the same changes to airbyte-ci which will allow publishing airbyte-lib and connectors to PyPi.

Base automatically changed from aj/airbyte-lib-caches-base to master January 12, 2024 17:43
@aaronsteers
Copy link
Collaborator Author

Closing. See this comment from above for spin-off PRs:

#33878 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants