Source Mixpanel: add insert_id to export stream schema to have a unique event identifier #21837

jeremySrgt · 2023-01-25T10:39:04Z

What

This change add the "insert_id" Mixpanel's property to the default schema of the export stream. Since the insert_id field act as a unique identifier to an event, we need it to perform deduplication. Related issue #21460

How

Adding insert_id to the export schema

  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "event": {
      "type": ["null", "string"]
    },
    "distinct_id": {
      "type": ["null", "string"]
    },
    "insert_id": {
      "type": ["null", "string"]
    },
    "time": {
      "type": ["null", "string"],
      "format": "date-time"
    },
    "labels": {
      "type": ["null", "array"],
      "items": {
        "type": ["null", "string"]
      }
    },
    "sampling_factor": {
      "type": ["null", "integer"]
    },
    "dataset": {
      "type": ["null", "string"]
    }
  }
}

🚨 User Impact 🚨

No breaking change, just a new available field when running a sync with the Mixpanel source

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (0.78s):
      27 passed

Integration

cachedir: .pytest_cache
hypothesis profile 'default' -> configfile: pytest.ini
plugins: hypothesis-6.54.6, requests-mock-1.9.3, mock-3.6.1, sugar-0.9.6, timeout-1.4.2, cov-3.0.0
collecting ... 

Results (0.28s):

Acceptance

Put your acceptance tests output here.

CLAassistant · 2023-01-25T10:39:13Z

All committers have signed the CLA.

…ream

marcosmarxm · 2023-04-11T20:50:56Z

/test connector=connectors/source-mixpanel

🕑 connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/4671893317
✅ connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/4671893317
Python tests coverage:

Name                                         Stmts   Miss  Cover
----------------------------------------------------------------
source_mixpanel/utils.py                         8      0   100%
source_mixpanel/streams/revenue.py              14      0   100%
source_mixpanel/streams/funnels.py              57      0   100%
source_mixpanel/streams/cohorts.py              15      0   100%
source_mixpanel/streams/__init__.py              9      0   100%
source_mixpanel/property_transformation.py      19      0   100%
source_mixpanel/__init__.py                      2      0   100%
source_mixpanel/streams/base.py                 93      3    97%
source_mixpanel/streams/export.py               68      3    96%
source_mixpanel/streams/engage.py               88      6    93%
source_mixpanel/source.py                       79      8    90%
source_mixpanel/streams/annotations.py          16      3    81%
source_mixpanel/streams/cohort_members.py       21      7    67%
source_mixpanel/testing.py                      29     11    62%
----------------------------------------------------------------
TOTAL                                          518     41    92%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
================== 41 passed, 1 skipped in 7803.28s (2:10:03) ==================

…ream

archangelic · 2023-04-14T15:19:59Z

/test connector=connectors/source-mixpanel

🕑 connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/4701330578
✅ connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/4701330578
Python tests coverage:

Name                                         Stmts   Miss  Cover
----------------------------------------------------------------
source_mixpanel/utils.py                         8      0   100%
source_mixpanel/streams/revenue.py              14      0   100%
source_mixpanel/streams/funnels.py              57      0   100%
source_mixpanel/streams/cohorts.py              15      0   100%
source_mixpanel/streams/__init__.py              9      0   100%
source_mixpanel/property_transformation.py      19      0   100%
source_mixpanel/__init__.py                      2      0   100%
source_mixpanel/streams/base.py                 93      3    97%
source_mixpanel/streams/export.py               68      3    96%
source_mixpanel/streams/engage.py               88      6    93%
source_mixpanel/source.py                       79      8    90%
source_mixpanel/streams/annotations.py          16      3    81%
source_mixpanel/streams/cohort_members.py       21      7    67%
source_mixpanel/testing.py                      29     11    62%
----------------------------------------------------------------
TOTAL                                          518     41    92%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
================= 41 passed, 1 skipped in 13508.35s (3:45:08) ==================

lazebnyi

LGTM!

marcosmarxm · 2023-05-09T17:15:34Z

Waiting CI get green again.

jeremySrgt · 2023-05-15T12:21:31Z

Waiting CI get green again.

@marcosmarxm Do i need to do somethings about it on my side ?

marcosmarxm · 2023-05-15T14:26:51Z

@jeremySrgt nope waiting our team to fix CI and run merge your contribution.

…ream

marcosmarxm · 2023-05-15T14:33:19Z

/publish connector=connectors/source-mixpanel

🕑 Publishing the following connectors:
connectors/source-mixpanel
https://github.com/airbytehq/airbyte/actions/runs/4981668677

Connector	Version	Did it publish?	Were definitions generated?
connectors/source-mixpanel	0.1.34	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

marcosmarxm · 2023-05-23T19:27:06Z

/test connector=connectors/source-mixpanel

🕑 connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/5061279988

marcosmarxm · 2023-05-26T13:45:32Z

The CI continue to fail for Mixpanel connect. Waiting the fix to merge this contribution.

…ream

marcosmarxm · 2023-06-05T14:52:51Z

/test connector=connectors/source-mixpanel

🕑 connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/5178603252
✅ connectors/source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/5178603252
Python tests coverage:

Name                                         Stmts   Miss  Cover
----------------------------------------------------------------
source_mixpanel/utils.py                         8      0   100%
source_mixpanel/streams/revenue.py              14      0   100%
source_mixpanel/streams/funnels.py              57      0   100%
source_mixpanel/streams/cohorts.py              15      0   100%
source_mixpanel/streams/__init__.py              9      0   100%
source_mixpanel/property_transformation.py      19      0   100%
source_mixpanel/__init__.py                      2      0   100%
source_mixpanel/streams/base.py                 90      3    97%
source_mixpanel/streams/export.py               76      5    93%
source_mixpanel/streams/engage.py               88      6    93%
source_mixpanel/source.py                       79      8    90%
source_mixpanel/streams/annotations.py          16      3    81%
source_mixpanel/streams/cohort_members.py       21      7    67%
source_mixpanel/testing.py                      29     11    62%
----------------------------------------------------------------
TOTAL                                          523     43    92%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:598: The previous and actual discovered catalogs are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:695: This tests currently leads to too much failures. We need to fix the connectors at scale first.
================== 40 passed, 3 skipped in 7803.53s (2:10:03) ==================

jeremySrgt added 2 commits January 25, 2023 11:18

Add Mixpanel's property "insert_id" to export schema

f70602b

Bump Mixpanel connector version

7dcdb0f

octavia-squidington-iv added area/connectors Connector related issues bounty community connectors/source/mixpanel labels Jan 25, 2023

Add changelog to source readme

43fbf83

octavia-squidington-iv added the area/documentation Improvements or additions to documentation label Jan 25, 2023

jeremySrgt changed the title ~~Source Mixpanel: add insert_id to export stream to have a unique event identifier~~ Source Mixpanel: add insert_id to export stream schema to have a unique event identifier Jan 25, 2023

jeremySrgt mentioned this pull request Jan 25, 2023

Source Mixpanel: Missing "insert_id" field on the Mixpanel export stream normalized data #21460

Closed

jeremySrgt added 2 commits February 15, 2023 09:58

Merge branch 'master' into mixpanel_add_unique_insert_id_to_export_st…

e71d505

…ream

Merge branch 'master' into mixpanel_add_unique_insert_id_to_export_st…

e4562a8

…ream

marcosmarxm added the ga-connector-change label Apr 7, 2023

marcosmarxm assigned archangelic Apr 7, 2023

Merge branch 'master' into mixpanel_add_unique_insert_id_to_export_st…

f703680

…ream

archangelic and others added 3 commits April 13, 2023 08:48

Merge branch 'master' into mixpanel_add_unique_insert_id_to_export_st…

ffbdc51

…ream

Merge branch 'master' into mixpanel_add_unique_insert_id_to_export_st…

f129663

…ream

update version

cc30fad

archangelic approved these changes Apr 14, 2023

View reviewed changes

marcosmarxm self-assigned this Apr 17, 2023

marcosmarxm added the gl-review label Apr 17, 2023

marcosmarxm requested a review from lazebnyi April 25, 2023 13:40

lazebnyi approved these changes Apr 25, 2023

View reviewed changes

lazebnyi self-assigned this May 5, 2023

lazebnyi added the team/connectors-python label May 5, 2023

marcosmarxm unassigned archangelic May 12, 2023

marcosmarxm removed their assignment May 12, 2023

marcosmarxm added community-legacy and removed community labels May 15, 2023

marcosmarxm added 2 commits May 15, 2023 11:28

Merge branch 'master' into mixpanel_add_unique_insert_id_to_export_st…

1f44aec

…ream

Update metadata.yaml

084e3a6

solve conflict

aea1db3

marcosmarxm added community and removed community-legacy labels May 26, 2023

Merge branch 'master' into mixpanel_add_unique_insert_id_to_export_st…

2baca51

…ream

marcosmarxm merged commit d369fd9 into airbytehq:master Jun 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Mixpanel: add insert_id to export stream schema to have a unique event identifier #21837

Source Mixpanel: add insert_id to export stream schema to have a unique event identifier #21837

jeremySrgt commented Jan 25, 2023 •

edited

Loading

CLAassistant commented Jan 25, 2023 •

edited

Loading

marcosmarxm commented Apr 11, 2023 •

edited by github-actions bot

Loading

archangelic commented Apr 14, 2023 •

edited by github-actions bot

Loading

lazebnyi left a comment

marcosmarxm commented May 9, 2023

jeremySrgt commented May 15, 2023

marcosmarxm commented May 15, 2023

marcosmarxm commented May 15, 2023 •

edited by github-actions bot

Loading

marcosmarxm commented May 23, 2023 •

edited by github-actions bot

Loading

marcosmarxm commented May 26, 2023

marcosmarxm commented Jun 5, 2023 •

edited by github-actions bot

Loading

Source Mixpanel: add insert_id to export stream schema to have a unique event identifier #21837

Source Mixpanel: add insert_id to export stream schema to have a unique event identifier #21837

Conversation

jeremySrgt commented Jan 25, 2023 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

CLAassistant commented Jan 25, 2023 • edited Loading

marcosmarxm commented Apr 11, 2023 • edited by github-actions bot Loading

Build Passed

archangelic commented Apr 14, 2023 • edited by github-actions bot Loading

Build Passed

lazebnyi left a comment

Choose a reason for hiding this comment

marcosmarxm commented May 9, 2023

jeremySrgt commented May 15, 2023

marcosmarxm commented May 15, 2023

marcosmarxm commented May 15, 2023 • edited by github-actions bot Loading

marcosmarxm commented May 23, 2023 • edited by github-actions bot Loading

marcosmarxm commented May 26, 2023

marcosmarxm commented Jun 5, 2023 • edited by github-actions bot Loading

Build Passed

jeremySrgt commented Jan 25, 2023 •

edited

Loading

CLAassistant commented Jan 25, 2023 •

edited

Loading

marcosmarxm commented Apr 11, 2023 •

edited by github-actions bot

Loading

archangelic commented Apr 14, 2023 •

edited by github-actions bot

Loading

marcosmarxm commented May 15, 2023 •

edited by github-actions bot

Loading

marcosmarxm commented May 23, 2023 •

edited by github-actions bot

Loading

marcosmarxm commented Jun 5, 2023 •

edited by github-actions bot

Loading