🎉 Source Amplitude: added missing attrs in events schema, enabled default availability strategy #25842

midavadim · 2023-05-05T13:31:17Z

added missing attrs in events schema,
enabled default avilability strategy

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user?

For connector PRs, use this section to explain which type of semantic versioning bump occurs as a result of the changes. Refer to our Semantic Versioning for Connectors guidelines for more information. Breaking changes to connectors must be documented by an Airbyte engineer (PR author, or reviewer for community PRs) by using the Breaking Change Release Playbook.

If there are breaking changes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Actions

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Community member? Grant edit access to maintainers (instructions)
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Connector version is set to 0.0.1
- Dockerfile has version 0.0.1
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- docs/integrations/<source or destination>/<name>.md including changelog with an entry for the initial version. See changelog example
- docs/integrations/README.md

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Unit & integration tests added

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

…ailability strategy

CLAassistant · 2023-05-05T13:31:23Z

All committers have signed the CLA.

github-actions · 2023-05-05T13:31:35Z

midavadim · 2023-05-05T13:34:03Z

/test connector=connectors/source-amplitude

🕑 connectors/source-amplitude https://github.com/airbytehq/airbyte/actions/runs/4893992230
✅ connectors/source-amplitude https://github.com/airbytehq/airbyte/actions/runs/4893992230
Python tests coverage:

Name                             Stmts   Miss  Cover
----------------------------------------------------
source_amplitude/components.py      68      0   100%
source_amplitude/__init__.py         2      0   100%
source_amplitude/source.py          19      9    53%
source_amplitude/streams.py        115     79    31%
----------------------------------------------------
TOTAL                              204     88    57%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
============ 39 passed, 1 skipped, 1 warning in 1090.35s (0:18:10) =============

midavadim · 2023-05-05T13:50:31Z

airbyte-integrations/connectors/source-amplitude/source_amplitude/streams.py

@@ -68,10 +67,6 @@ def url_base(self) -> str:
        subdomain = "analytics.eu." if self.data_region == "EU Residency Server" else ""
        return f"https://{subdomain}amplitude.com/api/"

-    @property


Connector does have custom error handling.
But we needed to handle such errors during the whole process of read_records, not just at the begining of the sync.

…lability-strategy

artem1205

availability strategy was removed, because for this stream API may return 404 (see docs ) during read of first slice.

midavadim · 2023-05-05T15:51:49Z

availability strategy was removed, because for this stream API may return 404 (see docs ) during read of first slice.

@artem1205
404 error is caught at line 163 and stream just returns empty list of records.
So this error does not get to 'availability strategy' handler

artem1205 · 2023-05-05T15:56:02Z

Yep, but this means that availability strategy is useless here, isn't it?

midavadim · 2023-05-05T15:57:32Z

Yep, but this means that availability strategy is useless here, isn't it?

I suppose it can handle another types of errors, what do you think?

erohmensing

Nice work! Could you please also add the missing column declarations to the cohorts stream? Here are all of the missing columns from the past week (they look pretty much teh same last month)

This comes from the "please check the datadog dashboard" note here. I guess our test doesn't catch it because cohorts is declared as an empty stream.

erohmensing · 2023-05-09T14:01:41Z

airbyte-integrations/connectors/source-amplitude/acceptance-test-config.yml

@@ -30,7 +30,6 @@ acceptance_tests:
          extra_fields: no
          exact_order: no
          extra_records: yes
-        fail_on_extra_columns: false


erohmensing · 2023-05-09T14:09:08Z

airbyte-integrations/connectors/source-amplitude/source_amplitude/streams.py

-    def availability_strategy(self) -> Optional["AvailabilityStrategy"]:
-        return None


Just double checked - I think @midavadim is right. The availability strategy's get_first_record_for_slice calls stream.read_records so the 404 error handling that returns empty records should work, and the AS should say "connected to stream, but got 0 records". So this should work in that case 👍🏻

maxi297

LGTM given Ella's concern is addressed!

…t-and-avialability-strategy' into midavadim/24941-amplitude-fix-sat-and-avialability-strategy

…lability-strategy

midavadim · 2023-05-09T19:30:05Z

@erohmensing

I have added missing fields for cohort stream.
Unfortunately, I do not have real data in that stream to check if the real types are the same.
Types for added fields are taken from API doc:
https://www.docs.developers.amplitude.com/analytics/apis/behavioral-cohorts-api/#get-all-cohorts-response

erohmensing · 2023-05-09T20:53:11Z

@midavadim that's fine with me! If any of the types are wrong, that's on the documentation, not us. And we'll catch them in the schema validation errors. Thank you! ⭐

erohmensing · 2023-05-09T20:54:10Z

Then I think we can safely close #24480 as well :)

midavadim · 2023-05-09T21:32:17Z

/publish connector=connectors/source-amplitude

🕑 Publishing the following connectors:
connectors/source-amplitude
https://github.com/airbytehq/airbyte/actions/runs/4930680711

Connector	Version	Did it publish?	Were definitions generated?
connectors/source-amplitude	0.2.4	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

…lability-strategy

…ault availability strategy (airbytehq#25842) * Added missing attrs in events schema, updated SAT, enabled default availability strategy * updated connector version * added missing fields for cohort stream * auto-bump connector version --------- Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

Added missing attrs in events schema, updated SAT, enabled default av…

0652efa

…ailability strategy

octavia-squidington-iii added the area/connectors Connector related issues label May 5, 2023

octavia-squidington-iii added the connectors/source/amplitude label May 5, 2023

midavadim changed the title ~~🎉 Source Amplitude: added missing attrs in events schema, enabled default avilability strategy~~ 🎉 Source Amplitude: added missing attrs in events schema, enabled default availability strategy May 5, 2023

updated connector version

55e42fd

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label May 5, 2023

midavadim requested a review from artem1205 May 5, 2023 13:45

This was linked to issues May 5, 2023

Source Amplitude: Add undeclared columns to spec #24480

Closed

Amplitude Improvement Epic #24941

Closed

Source source-amplitude: implement AvailabilityStrategy #22248

Closed

midavadim commented May 5, 2023

View reviewed changes

Merge branch 'master' into midavadim/24941-amplitude-fix-sat-and-avia…

d076d5b

…lability-strategy

artem1205 requested changes May 5, 2023

View reviewed changes

artem1205 approved these changes May 5, 2023

View reviewed changes

midavadim requested a review from maxi297 May 5, 2023 18:50

lazebnyi requested a review from erohmensing May 8, 2023 12:59

erohmensing reviewed May 9, 2023

View reviewed changes

maxi297 approved these changes May 9, 2023

View reviewed changes

midavadim added 3 commits May 9, 2023 22:25

added missing fields for cohort stream

9ffaec6

Merge remote-tracking branch 'origin/midavadim/24941-amplitude-fix-sa…

5eb7f9c

…t-and-avialability-strategy' into midavadim/24941-amplitude-fix-sat-and-avialability-strategy

Merge branch 'master' into midavadim/24941-amplitude-fix-sat-and-avia…

1f9ef31

…lability-strategy

midavadim requested a review from erohmensing May 9, 2023 20:43

erohmensing approved these changes May 9, 2023

View reviewed changes

midavadim and others added 2 commits May 10, 2023 00:32

Merge branch 'master' into midavadim/24941-amplitude-fix-sat-and-avia…

b50a12d

…lability-strategy

auto-bump connector version

2bf0ce0

midavadim enabled auto-merge (squash) May 9, 2023 22:21

midavadim added 2 commits May 10, 2023 01:21

Merge branch 'master' into midavadim/24941-amplitude-fix-sat-and-avia…

c425afb

…lability-strategy

Merge branch 'master' into midavadim/24941-amplitude-fix-sat-and-avia…

fdf4dfb

…lability-strategy

midavadim merged commit f5321d1 into master May 10, 2023
26 of 27 checks passed

midavadim deleted the midavadim/24941-amplitude-fix-sat-and-avialability-strategy branch May 10, 2023 08:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source Amplitude: added missing attrs in events schema, enabled default availability strategy #25842

🎉 Source Amplitude: added missing attrs in events schema, enabled default availability strategy #25842

midavadim commented May 5, 2023

CLAassistant commented May 5, 2023 •

edited

github-actions bot commented May 5, 2023 •

edited by midavadim

midavadim commented May 5, 2023 •

edited by github-actions bot

midavadim May 5, 2023

artem1205 left a comment

midavadim commented May 5, 2023

artem1205 commented May 5, 2023

midavadim commented May 5, 2023

erohmensing left a comment

erohmensing May 9, 2023

erohmensing May 9, 2023

maxi297 left a comment

midavadim commented May 9, 2023

erohmensing commented May 9, 2023

erohmensing commented May 9, 2023

midavadim commented May 9, 2023 •

edited by github-actions bot

		def availability_strategy(self) -> Optional["AvailabilityStrategy"]:
		return None

🎉 Source Amplitude: added missing attrs in events schema, enabled default availability strategy #25842

🎉 Source Amplitude: added missing attrs in events schema, enabled default availability strategy #25842

Conversation

midavadim commented May 5, 2023

🚨 User Impact 🚨

Pre-merge Actions

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

CLAassistant commented May 5, 2023 • edited

github-actions bot commented May 5, 2023 • edited by midavadim

Before Merging a Connector Pull Request

midavadim commented May 5, 2023 • edited by github-actions bot

Build Passed

midavadim May 5, 2023

Choose a reason for hiding this comment

artem1205 left a comment

Choose a reason for hiding this comment

midavadim commented May 5, 2023

artem1205 commented May 5, 2023

midavadim commented May 5, 2023

erohmensing left a comment

Choose a reason for hiding this comment

erohmensing May 9, 2023

Choose a reason for hiding this comment

erohmensing May 9, 2023

Choose a reason for hiding this comment

maxi297 left a comment

Choose a reason for hiding this comment

midavadim commented May 9, 2023

erohmensing commented May 9, 2023

erohmensing commented May 9, 2023

midavadim commented May 9, 2023 • edited by github-actions bot

CLAassistant commented May 5, 2023 •

edited

github-actions bot commented May 5, 2023 •

edited by midavadim

midavadim commented May 5, 2023 •

edited by github-actions bot

midavadim commented May 9, 2023 •

edited by github-actions bot