Repository navigation
Migrate Shopify Streams from REST API to GraphQL API - #201
Merged
Merged
Conversation
…tafields_customers and metafields_orders
sgandhi1311
changed the title
sgandhi1311
marked this pull request as ready for review
March 11, 2025 04:54
sgandhi1311
force-pushed
the
TDL-26979-gql-query-schema-update
branch
from
March 19, 2025 10:41
4be5461 to
fc706be
Compare
sgandhi1311
force-pushed
the
TDL-26979-gql-query-schema-update
branch
from
March 19, 2025 10:42
fc706be to
71619f3
Compare
* graphql implementataion for custoemrs, events and locations stream * Use graphql endpoint for orders * add the queries in the respective stream class * Update the logic for inventory levels stream * Optimize inventory levels logic * update transactions and order_refunds logic * Initial commit for abandoned checkouts and collections streams * Remove reference for the collects and custom collections * optimize the order_refunds and transactions stream logic * partial logic for collections * update locations query * add collections streams changes and separate out metafields as individual streams * remove gql query file * remove gql query file reference and include the query in the code * fix abandoned checkouts logic * format the code * remove metafields reference from init file * update checkout schema * fix typo * set the default page size to 250 * make pylint happy and format the code * upgrade python packages * fix circleci pylint issues * modify variable name in collections stream * optimize get query params method * move the graphql base code to the base file * removed reference to the gql_base file * fix the pylint * optimize the source code * update unit test cases * update exception in unit test * Refactor the code - 1. Use the single query for inventiry levels to fetch the child object as well. 2. Add Sortkey in the products query 3. Update query names for the respective metafields * move date_window_size to the parent class * make pylint happy * make pylint happy * Make metafields related stream, order_refunds and transactions stream as pseudo incremental * make pylint happy * the similar scencario is tested in the integration test, so we can skip this test. * Update Integration Tests for GraphQL Migration (#205) * make initial base file changes * add individual meta streams * update all fields integration test * update locations replication key in test suite * remove old references of the stream * avoid testing on the events stream * reduce the default page size for metafields * update state file in the bookmarks * include events in the expected streams * fix interrupt sync * update foreign key for transactions table
sgandhi1311
changed the title
prijendev
approved these changes
Mar 28, 2025
|
Hey guys - StitchData customer here. It looks like the GraphQL rewrite omitted the These two issues have broken our reporting pipelines and made us unable to really use StitchData for our usecase. Is this GraphQL rewrite still a work-in-progress? Feels like a lot was missed here and StitchData won't let us roll back or use the old version. At the very least, the new GraphQL-based API call should have all the same data. Should I submit a GitHub issue for this, or is it something we need to ask StitchData support about? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of change
This PR migrates several streams in tap-shopify from the REST API to the GraphQL API, improving performance and efficiency. Additionally, some stream changes were made to better align with Shopify’s data structure.
Changes & Updates
Migrated Streams (REST → GraphQL)
Metafields Stream Refactoring
Removed: metafields (generic metafields stream)
Introduced New Streams:
Collections Stream Refactoring
Removed: collects and custom_collections
Introduced New Stream: collections (consolidated collection stream)
QA steps
Risks
Rollback steps
AI generated code
https://internal.qlik.dev/general/ways-of-working/code-reviews/#guidelines-for-ai-generated-code