🎉 Source Mixpanel: Beta preparation

[roman-romanov-o]

What

1. Acceptances tests were failing owing to changed ratelimit (60 requests per hour for any endpoint)
2. Ratelimit issue & https://github.com/airbytehq/alpha-beta-issues/issues/70
3. High volume issue
4. Normaliztion failed for custom Export
5. Move streams to distinct module to reduce file size
6. Update doc

How

1. Added decorator for streams, that check is_testing flag in config, and reduces amount of requests that is made in acceptance tests
2. Reduce ratelimit for all connectors
3. Add date_window_size to spec, so client will be able to iterate through data via smaller batches

Sort property names, and check for such cases, and transform userName -> _userName

If such collission occurs more than one time, then

userName
UserName
username

will be transformed to

userName
_UserName
__username

Recommended reading order

Huge line change number is caused only by moving streams into distinct module

• Only real change in source.py is added decorator to streams method
• Only real change in sources is updated Export (streams/export.py) source, where properties are transformed via transform_property_names

Pre-merge Checklist

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.
• /test connector=connectors/<name> command is passing
• New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (master@474349e). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #13372   +/-   ##
=========================================
  Coverage          ?   90.49%           
=========================================
  Files             ?       13           
  Lines             ?      421           
  Branches          ?        0           
=========================================
  Hits              ?      381           
  Misses            ?       40           
  Partials          ?        0           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 474349e...935ef87. Read the comment docs.

[roman-romanov-o]

/test connector=source-mixpanel

🕑 source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2422458061
✅ source-mixpanel https://github.com/airbytehq/airbyte/actions/runs/2422458061
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        77      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/tests/test_core.py              294    106    64%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  960    246    74%
Name                                         Stmts   Miss  Cover
----------------------------------------------------------------
source_mixpanel/streams/revenue.py              14      0   100%
source_mixpanel/streams/funnels.py              56      0   100%
source_mixpanel/streams/annotations.py           6      0   100%
source_mixpanel/streams/__init__.py              9      0   100%
source_mixpanel/property_transformation.py      19      0   100%
source_mixpanel/__init__.py                      2      0   100%
source_mixpanel/source.py                       45      1    98%
source_mixpanel/streams/engage.py               89      5    94%
source_mixpanel/streams/base.py                 71      7    90%
source_mixpanel/streams/cohorts.py              20      3    85%
source_mixpanel/streams/export.py               46      8    83%
source_mixpanel/streams/cohort_members.py       19      5    74%
source_mixpanel/testing.py                      25     11    56%
----------------------------------------------------------------
TOTAL                                          421     40    90%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:23: `future_state_path` not specified, skipping
================== 23 passed, 1 skipped in 400.89s (0:06:40) ===================

[lazebnyi]

LGTM

[supertopher]

@bazarnov This PR is breaking production builds because it has not been published

stack trace

2022-06-03 12:54:33 INFO i.a.c.s.SeedConnectorSpecGenerator(fetchSpecFromGCS):119 - Seeded spec not found for docker image airbyte/source-mixpanel:0.1.17 - fetching from GCS bucket io-airbyte-cloud-spec-cache...
Exception in thread "main" java.lang.RuntimeException: Failed to fetch valid spec file for docker image airbyte/source-mixpanel:0.1.17 from GCS bucket io-airbyte-cloud-spec-cache. This will continue to fail until the connector change has been approved and published. See https://github.com/airbytehq/airbyte/tree/master/docs/connector-development#publishing-a-connector for more details.
        at io.airbyte.config.specs.SeedConnectorSpecGenerator.lambda$fetchSpecFromGCS$4(SeedConnectorSpecGenerator.java:121)
        at java.base/java.util.Optional.orElseThrow(Optional.java:403)
        at io.airbyte.config.specs.SeedConnectorSpecGenerator.fetchSpecFromGCS(SeedConnectorSpecGenerator.java:121)
        at io.airbyte.config.specs.SeedConnectorSpecGenerator.lambda$fetchUpdatedSeedSpecs$3(SeedConnectorSpecGenerator.java:114)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
        at io.airbyte.config.specs.SeedConnectorSpecGenerator.fetchUpdatedSeedSpecs(SeedConnectorSpecGenerator.java:115)
        at io.airbyte.config.specs.SeedConnectorSpecGenerator.run(SeedConnectorSpecGenerator.java:83)
        at io.airbyte.config.specs.SeedConnectorSpecGenerator.main(SeedConnectorSpecGenerator.java:73)

> Task :airbyte-config:specs:generateSeedConnectorSpecs FAILED

I honestly don't really understand the workflow or what publishing is exactly in this use case. Can you help me understand the best way to get prod back in fighting shape quickly?