[faker] decouple stream state #20492

evantahler · 2022-12-14T20:10:33Z

There was a hacky implementation using global variables before this PR. Now each stream can work independently.

~~This /would/ be a breaking change for a normal connector (as the format of STATE is changing), but it's faker, so 🤷‍♂️~~ Let's do it right!

evantahler · 2022-12-15T19:45:56Z

/test connector=connectors/source-faker

🕑 connectors/source-faker https://github.com/airbytehq/airbyte/actions/runs/3707419735
✅ connectors/source-faker https://github.com/airbytehq/airbyte/actions/runs/3707419735
Python tests coverage:

Name                       Stmts   Miss  Cover
----------------------------------------------
source_faker/utils.py         11      0   100%
source_faker/__init__.py       2      0   100%
source_faker/streams.py      166      3    98%
source_faker/source.py        16      3    81%
----------------------------------------------
TOTAL                        195      6    97%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1603    336    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:386: Backward compatibility tests are disabled for version 1.0.0.
======================== 29 passed, 2 skipped in 36.55s ========================

alafanechere

Minor clarification request about the cursor field declaration

airbyte-integrations/connectors/source-faker/source_faker/streams.py

docs/integrations/sources/faker.md

airbyte-integrations/connectors/source-faker/Dockerfile

airbyte-integrations/connectors/source-faker/acceptance-test-config.yml

Co-authored-by: Augustin <augustin@airbyte.io>

evantahler · 2022-12-17T01:34:18Z

/test connector=connectors/source-faker

🕑 connectors/source-faker https://github.com/airbytehq/airbyte/actions/runs/3717832766
✅ connectors/source-faker https://github.com/airbytehq/airbyte/actions/runs/3717832766
Python tests coverage:

Name                       Stmts   Miss  Cover
----------------------------------------------
source_faker/utils.py         11      0   100%
source_faker/__init__.py       2      0   100%
source_faker/streams.py      166      2    99%
source_faker/source.py        16      3    81%
----------------------------------------------
TOTAL                        195      5    97%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1603    336    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:386: Backward compatibility tests are disabled for version 1.0.0.
======================== 29 passed, 2 skipped in 34.59s ========================

* Faker CSV Streaming utilities * readme * don't do a final pipe to jq or you will run out or ram * doc

* Faker is 250% faster * threads in spec + lint * pass tests * revert changes to record helper * cleanup * update expected_records * bump default records-per-slice to 1k * enforce unique email addresses * cleanup * more comments * `parallelism` and pass tests * update expected records * cleanup notes

evantahler · 2023-01-02T22:43:16Z

/test connector=connectors/source-faker

🕑 connectors/source-faker https://github.com/airbytehq/airbyte/actions/runs/3825523945
✅ connectors/source-faker https://github.com/airbytehq/airbyte/actions/runs/3825523945
Python tests coverage:

Name                                               Stmts   Miss  Cover
----------------------------------------------------------------------
source_faker/__init__.py                               2      0   100%
source_faker/streams.py                              126      1    99%
source_faker/source.py                                17      3    82%
source_faker/utils.py                                 18      6    67%
source_faker/airbyte_message_with_cached_json.py       8      4    50%
source_faker/user_generator.py                        28     16    43%
source_faker/purchase_generator.py                    55     41    25%
----------------------------------------------------------------------
TOTAL                                                254     71    72%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1603    336    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:386: Backward compatibility tests are disabled for version 1.0.0.
======================== 30 passed, 1 skipped in 39.15s ========================

evantahler · 2023-01-02T22:58:30Z

/publish connector=connectors/source-faker

🕑 Publishing the following connectors:
connectors/source-faker
https://github.com/airbytehq/airbyte/actions/runs/3825568063

Connector	Did it publish?	Were definitions generated?
connectors/source-faker	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

evantahler · 2023-01-03T00:03:54Z

/approve-and-merge reason="the platform tests are failing but this is a connector-only change. Also, the connector was already published"

octavia-approvington · 2023-01-03T00:04:41Z

A crack team of mammals has made a decision.

evantahler added 2 commits December 14, 2022 12:09

[faker] decouple stream state

4fd3ba8

add PR #

5ab75ff

octavia-squidington-iv added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/faker labels Dec 14, 2022

evantahler added 3 commits December 14, 2022 13:20

commit Stream instantiate changes

5333c1b

fixup expected record

5466f31

skip backward test for this version too

5ca497f

evantahler requested review from sherifnada and alafanechere December 16, 2022 00:15

evantahler marked this pull request as ready for review December 16, 2022 00:16

alafanechere requested changes Dec 16, 2022

View reviewed changes

evantahler and others added 2 commits December 16, 2022 14:18

Apply suggestions from code review

fb408a8

Co-authored-by: Augustin <augustin@airbyte.io>

lint

2a967ae

evantahler requested a review from alafanechere December 19, 2022 19:03

evantahler added 2 commits December 19, 2022 11:03

Merge branch 'master' into evan/faker-hydra

fcab572

Merge branch 'master' into evan/faker-hydra

b2b94dd

alafanechere approved these changes Dec 20, 2022

View reviewed changes

evantahler added 6 commits December 21, 2022 15:37

Merge branch 'master' into evan/faker-hydra

fbdff95

Create realistic datasets of 10GB, 100GB, and 1TB in size (#20558)

f178b14

* Faker CSV Streaming utilities * readme * don't do a final pipe to jq or you will run out or ram * doc

Merge branch 'master' into evan/faker-hydra

84f8d2e

update readme

9ad7e04

update expected records

32c3773

auto-bump connector version

5604953

octavia-squidington-iii temporarily deployed to more-secrets January 2, 2023 23:19 — with GitHub Actions Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 2, 2023 23:20 — with GitHub Actions Inactive

octavia-approvington approved these changes Jan 3, 2023

View reviewed changes

octavia-approvington merged commit 2022f7d into master Jan 3, 2023

octavia-approvington deleted the evan/faker-hydra branch January 3, 2023 00:04

This was referenced Jan 6, 2023

Bump Airbyte version from 0.40.26 to 0.40.27 #21092

Closed

Bump Airbyte version from 0.40.26 to 0.40.27 #21121

Closed

github-actions bot mentioned this pull request Jan 6, 2023

Bump helm chart version reference to 0.43.15 #21125

Closed

This was referenced Jan 6, 2023

Bump Airbyte version from 0.40.26 to 0.40.27 #21130

Closed

Bump Airbyte version from 0.40.26 to 0.40.27 #21135

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[faker] decouple stream state #20492

[faker] decouple stream state #20492

evantahler commented Dec 14, 2022 •

edited

Loading

evantahler commented Dec 15, 2022 •

edited by github-actions bot

Loading

alafanechere left a comment

evantahler commented Dec 17, 2022 •

edited by github-actions bot

Loading

evantahler commented Jan 2, 2023 •

edited by github-actions bot

Loading

evantahler commented Jan 2, 2023 •

edited by github-actions bot

Loading

evantahler commented Jan 3, 2023

octavia-approvington commented Jan 3, 2023

[faker] decouple stream state #20492

[faker] decouple stream state #20492

Conversation

evantahler commented Dec 14, 2022 • edited Loading

evantahler commented Dec 15, 2022 • edited by github-actions bot Loading

Build Passed

alafanechere left a comment

Choose a reason for hiding this comment

evantahler commented Dec 17, 2022 • edited by github-actions bot Loading

Build Passed

evantahler commented Jan 2, 2023 • edited by github-actions bot Loading

Build Passed

evantahler commented Jan 2, 2023 • edited by github-actions bot Loading

evantahler commented Jan 3, 2023

octavia-approvington commented Jan 3, 2023

evantahler commented Dec 14, 2022 •

edited

Loading

evantahler commented Dec 15, 2022 •

edited by github-actions bot

Loading

evantahler commented Dec 17, 2022 •

edited by github-actions bot

Loading

evantahler commented Jan 2, 2023 •

edited by github-actions bot

Loading

evantahler commented Jan 2, 2023 •

edited by github-actions bot

Loading