🎉 Source Instagram: Improve, refactor STATE management
#27908
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
octavia-squidington-iii
added
area/connectors
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
octavia-squidington-iii
added
the
area/documentation
This was
linked to
issues
Jul 3, 2023
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/source-instagram/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| Connector version increment check | ✅ |
| QA checks | ✅ |
| Code format checks | ✅ |
| Connector package install | ✅ |
| Build source-instagram docker image for platform linux/x86_64 | ✅ |
| Unit tests | ✅ |
| Integration tests | ✅ |
| Acceptance tests | ✅ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-instagram test
lazebnyi
requested changes
Jul 5, 2023
airbyte-integrations/connectors/source-instagram/source_instagram/streams.py
Outdated
Show resolved
Hide resolved
…m-fix-noo-end-time-users-insights
lazebnyi
approved these changes
Jul 5, 2023
|
| Step | Result |
|---|---|
| Validate airbyte-integrations/connectors/source-instagram/metadata.yaml | ✅ |
| Connector version semver check | ✅ |
| QA checks | ✅ |
| Code format checks | ✅ |
| Connector package install | ✅ |
| Build source-instagram docker image for platform linux/x86_64 | ✅ |
| Unit tests | ✅ |
| Integration tests | ✅ |
| Acceptance tests | ✅ |
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-instagram test
jbfbell
pushed a commit
that referenced
this pull request
Jul 5, 2023
octavia-approvington
pushed a commit
that referenced
this pull request
Jul 6, 2023
* Adds data as JsonNode to pass through, running into memory issues so add JVM args to attach VisualVM * JsonNode * Lowers the optimal batch size to see if this improvements movement * Fixes NPE by checking if PartialAirbyteMessage contains a PartialAirbyteRecord * Fixes config switch when config is not explicitly set (config migration needed) * Adds logic to check if queue has elements before getting timeOfLastMessage * Add PartialSerialisedMessage test. (#27452) * Test deserialise. * Add tests. * Simplify and fix tests. * Format. * Adds tests for deserializeAirbyteMessage * Adds tests for deserializeAirbyteMessage with bad data * Cleans up deserializeAirbyteMessage and throws Exception when invalid message * More code cleanup --------- Co-authored-by: ryankfu <ryan.fu@airbyte.io> * 🤖 Auto format destination-snowflake code [skip ci] * Cleans up code w/o JVM args & rebase * 🤖 Auto format destination-snowflake code [skip ci] * Adds breadcrumb on the STATE message deviation and where the deserialize/serialize is done to unpack * 🤖 Auto format destination-snowflake code [skip ci] * Adds back line formatter removed and comment describing rational for lower batchSize * Bumps Snowflake version and type checks * Added note to remove PartialAirbyteRecordMessage with low resource testing * Automated Commit - Format and Process Resources Changes * Fix issue with multiple namespaces in snowflake not writing to the correct staging schema * Fix issue with multiple namespaces in snowflake not writing to the correct staging schema * remove stage name maniuplating method * update readme * Source Stripe: update credit_notes expected records (#27941) * Source Zendesk Talk: update expected records (#27942) * Source Xero: update expected records (#27943) * Metadata: Persist Registry entries (#27766) * DNC * Update poetry * Update dagster * Apply partition * Get metadata entry * Use helpers * Write registry entry to appropriate location * Delete when registry removed * Update to use new file (broken) * Render registry from registry entries * Run format * Fix plural issue * Update to all metadata file blobs * Fix test * Update to all blobs * Add ignore validation error for version logic * Rename to max_run_request * Pedros review * Ella suggestions Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com> * Update airbyte-ci/connectors/metadata_service/orchestrator/orchestrator/assets/registry_entry.py Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com> * Update naming * Add tests for connector type and deletion * Test safe parse * Format --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com> Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com> * Fix Dagster Deploy Failure (#27955) * Add pydantic * Add pydantic to orchestration deploy pipeline * 🐛 Source Jira: update expected records (#27951) * Source Jira: update expected records * Update issues expected records * Source Zendesk Chat: update expected records (#27965) * 🐛 Source Pipedrive: update expected records (#27967) * 🐛 Source Pinterest: update expected records (#27964) * ✨ Source Amazon-Ads: Add streams for portfolios and sponsored brands v3 (#27607) * Add stream for sponsored brands v3 * Add new stream Portfolios * Source Google Search Console: added discover and googleNews to searchType (#27952) * added discover and googleNews to searchType * updated changelog * fixed types for streams * 🎉 Source Instagram: Improve, refactor `STATE` management (#27908) * add test for enabling * update versions * fix test * update other snowflake loading method types * remove standard --------- Co-authored-by: ryankfu <ryan.fu@airbyte.io> Co-authored-by: Davin Chia <davinchia@gmail.com> Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: ryankfu <ryankfu@users.noreply.github.com> Co-authored-by: Augustin <augustin@airbyte.io> Co-authored-by: Arsen Losenko <20901439+arsenlosenko@users.noreply.github.com> Co-authored-by: Ben Church <ben@airbyte.io> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@sers.noreply.github.com> Co-authored-by: Ella Rohm-Ensing <erohmensing@gmail.com> Co-authored-by: Anatolii Yatsuk <35109939+tolik0@users.noreply.github.com> Co-authored-by: Daryna Ishchenko <80129833+darynaishchenko@users.noreply.github.com> Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Resolving:
How
user_lifetime_insightsstream could return the records with missingend_timeorvaluekeysSTATEmanagement by migrating from theget_updated_state()method toIncrementalMixininheritanceunit tests🚨 User Impact 🚨
No impact is expected - no breaking changes.