🐛 Source Shopify: validate `shop` input (name only, reject urls) #25961

bazarnov · 2023-05-10T14:46:05Z

What

Resolving: #25026

How

Given the default URL-like input: https://airbyte-integration-test.myshopify.com/ is not an option for the shop, as there is tons of variations of URL-like input, the only option to validate the input is to reject the URL-like ones and leave only straight name from https://airbyte-integration-test.myshopify.com/ > airbyte-integration-test

added pattern that validates there is no link/URL-like string for shop in spec.json

🚨 User Impact 🚨

No impact is expected, since this is not a breaking change, all existing connections should work as they are.

For connector PRs, use this section to explain which type of semantic versioning bump occurs as a result of the changes. Refer to our Semantic Versioning for Connectors guidelines for more information. Breaking changes to connectors must be documented by an Airbyte engineer (PR author, or reviewer for community PRs) by using the Breaking Change Release Playbook.

If there are breaking changes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Actions

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Community member? Grant edit access to maintainers (instructions)
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Connector version is set to 0.0.1
- Dockerfile has version 0.0.1
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- docs/integrations/<source or destination>/<name>.md including changelog with an entry for the initial version. See changelog example
- docs/integrations/README.md

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Unit & integration tests added

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

github-actions · 2023-05-10T14:46:27Z

bazarnov · 2023-05-10T14:49:28Z

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/4938355504
❌ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/4938355504
🐛 https://gradle.com/s/d6f2tk54zajmo

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream shop:...
SKIPPED [3] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
============= 1 failed, 58 passed, 3 skipped in 1207.49s (0:20:07) =============

evantahler

Question about the regex, but otherwise 👍

evantahler · 2023-05-10T15:05:25Z

airbyte-integrations/connectors/source-shopify/source_shopify/spec.json

@@ -11,6 +11,7 @@
        "type": "string",
        "title": "Shopify Store",
        "description": "The name of your Shopify store found in the URL. For example, if your URL was https://NAME.myshopify.com, then the name would be 'NAME'.",
+        "pattern": "^(?!https:\/\/)(?!http:\/\/)(?!.*\\.myshopify\\.)(?!.*\\.myshopify).*",


👍 This is not a breaking change, and only an additional UI validation for the data the connector is asking for.

The current regex (validated with https://rubular.com/) allows company-name.shopify.com, which I don't think is desired (see https://rubular.com/r/pG2m72BxZLCxFb), and I don't think the "capture" statements are working. If the goal is only to allow NAME (per the description), perhaps the pattern should also ignore any . characters as well?

The regex now will validate whether or not the user inputs some URLs which we cannot actually process correctly, since there is no problem with having the shop name = my.shop.name.myshopify.com, where my.shop.name as the name we need. I'm also trying to check the Shopify restrictions for the shop names (specifically, what symbols are allowed), so we are aligned with their rules to match the shop name pattern effectively. No doubts - this solution is step one, I'm sure we will improve this in the future as well as will hit some edge cases, as always.

Thus, if the input string contains https' or myshopify` (+ variations with prefix and suffix dots) we are good to catch that user inputs something we cannot actually handle correctly, so extracting the name of the shop should be done in prior and the field should only accept the effective name, not URLs. or similar like not (https://my_store.myshopify.com/ or my_store.myshopify.com)
there are tons of variations that could be made here, in order to minimize this variation, we need to strict this field to something that could be easily processed from our side.

evantahler

Question about the regex, but otherwise 👍

…reject-url-input

bazarnov · 2023-05-10T20:51:07Z

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/4941219989
✅ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/4941219989
Python tests coverage:

Name                               Stmts   Miss  Cover
------------------------------------------------------
source_shopify/shopify_schema.py   10152      0   100%
source_shopify/auth.py                20      0   100%
source_shopify/__init__.py             2      0   100%
source_shopify/transform.py           58      3    95%
source_shopify/utils.py               96      7    93%
source_shopify/source.py             454     66    85%
source_shopify/graphql.py             47     40    15%
------------------------------------------------------
TOTAL                              10829    116    99%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [3] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
================== 59 passed, 3 skipped in 2683.33s (0:44:43) ==================

bazarnov · 2023-05-10T22:08:17Z

/publish connector=connectors/source-shopify

🕑 Publishing the following connectors:
connectors/source-shopify
https://github.com/airbytehq/airbyte/actions/runs/4941754967

Connector	Version	Did it publish?	Were definitions generated?
connectors/source-shopify	0.3.4	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

…bytehq#25961)

fixed

a2b0f3b

bazarnov added the team/connectors-python label May 10, 2023

bazarnov self-assigned this May 10, 2023

octavia-squidington-iii added area/connectors Connector related issues connectors/source/shopify labels May 10, 2023

updated changelog

83a7743

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label May 10, 2023

bazarnov requested review from grubberr, davydov-d, lazebnyi and a team May 10, 2023 14:59

evantahler approved these changes May 10, 2023

View reviewed changes

lazebnyi approved these changes May 10, 2023

View reviewed changes

bazarnov added 2 commits May 10, 2023 23:38

updated expected records for shop

d66f08b

Merge remote-tracking branch 'origin/master' into baz/source-shopify-…

8c729d1

…reject-url-input

bazarnov linked an issue May 10, 2023 that may be closed by this pull request

Shopify doesn't accept full store name URL #25026

Closed

auto-bump connector version

3cae28c

bazarnov merged commit da60731 into master May 11, 2023

bazarnov deleted the baz/source-shopify-reject-url-input branch May 11, 2023 08:11

marcosmarxm pushed a commit to natalia-miinto/airbyte that referenced this pull request Jun 8, 2023

🐛 Source Shopify: validate shop input (name only, reject urls) (air…

e87d947

…bytehq#25961)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Shopify: validate `shop` input (name only, reject urls) #25961

🐛 Source Shopify: validate `shop` input (name only, reject urls) #25961

bazarnov commented May 10, 2023 •

edited

Loading

github-actions bot commented May 10, 2023 •

edited by bazarnov

Loading

bazarnov commented May 10, 2023 •

edited by github-actions bot

Loading

evantahler left a comment

evantahler May 10, 2023

evantahler May 10, 2023

bazarnov May 10, 2023

bazarnov May 10, 2023

evantahler left a comment

bazarnov commented May 10, 2023 •

edited by github-actions bot

Loading

bazarnov commented May 10, 2023 •

edited by github-actions bot

Loading

🐛 Source Shopify: validate shop input (name only, reject urls) #25961

🐛 Source Shopify: validate shop input (name only, reject urls) #25961

Conversation

bazarnov commented May 10, 2023 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Actions

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

github-actions bot commented May 10, 2023 • edited by bazarnov Loading

Before Merging a Connector Pull Request

bazarnov commented May 10, 2023 • edited by github-actions bot Loading

Build Failed

evantahler left a comment

Choose a reason for hiding this comment

evantahler May 10, 2023

Choose a reason for hiding this comment

evantahler May 10, 2023

Choose a reason for hiding this comment

bazarnov May 10, 2023

Choose a reason for hiding this comment

bazarnov May 10, 2023

Choose a reason for hiding this comment

evantahler left a comment

Choose a reason for hiding this comment

bazarnov commented May 10, 2023 • edited by github-actions bot Loading

Build Passed

bazarnov commented May 10, 2023 • edited by github-actions bot Loading

🐛 Source Shopify: validate `shop` input (name only, reject urls) #25961

🐛 Source Shopify: validate `shop` input (name only, reject urls) #25961

bazarnov commented May 10, 2023 •

edited

Loading

github-actions bot commented May 10, 2023 •

edited by bazarnov

Loading

bazarnov commented May 10, 2023 •

edited by github-actions bot

Loading

bazarnov commented May 10, 2023 •

edited by github-actions bot

Loading

bazarnov commented May 10, 2023 •

edited by github-actions bot

Loading