Source S3: update block size for json by artem1205 · Pull Request #21210 · airbytehq/airbyte

Artem Inzhyyants (artem1205) · 2023-01-10T17:17:59Z

What

Resolving https://github.com/airbytehq/oncall/issues/1257
update block size for json

How

set block size for 0 as default value. Pyarrow crashes sometimes if block size is too small with notimplementederror

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Artem Inzhyyants (artem1205) · 2023-01-10T17:21:40Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/3885693467
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/3885693467
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   56-63, 68-81, 244
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/jsonl_spec.py                13      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/avro_spec.py                  5      0   100%
source_s3/s3file.py                                                  37      0   100%
source_s3/s3_utils.py                                                20      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source_files_abstract/storagefile.py                       23      1    96%
source_s3/stream.py                                                  43      3    93%
source_s3/source_files_abstract/stream.py                           242     17    93%
source_s3/source_files_abstract/formats/abstract_file_parser.py      41      4    90%
source_s3/source.py                                                  27      4    85%
source_s3/source_files_abstract/formats/csv_parser.py               102     30    71%
source_s3/source_files_abstract/file_info.py                         26      8    69%
source_s3/utils.py                                                   31     10    68%
source_s3/exceptions.py                                              10      4    60%
source_s3/source_files_abstract/source.py                            39     16    59%
source_s3/source_files_abstract/spec.py                              44     22    50%
source_s3/source_files_abstract/formats/jsonl_parser.py              53     32    40%
source_s3/source_files_abstract/formats/avro_parser.py               39     25    36%
source_s3/source_files_abstract/formats/parquet_parser.py            64     44    31%
-------------------------------------------------------------------------------------
TOTAL                                                               886    220    75%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/storagefile.py                       23      0   100%
source_s3/source_files_abstract/spec.py                              44      0   100%
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/jsonl_spec.py                13      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/avro_spec.py                  5      0   100%
source_s3/source.py                                                  27      0   100%
source_s3/s3file.py                                                  37      0   100%
source_s3/exceptions.py                                              10      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source_files_abstract/formats/parquet_parser.py            64      2    97%
source_s3/source_files_abstract/formats/abstract_file_parser.py      41      2    95%
source_s3/stream.py                                                  43      3    93%
source_s3/source_files_abstract/formats/avro_parser.py               39      3    92%
source_s3/source_files_abstract/formats/jsonl_parser.py              53      5    91%
source_s3/source_files_abstract/file_info.py                         26      3    88%
source_s3/source_files_abstract/source.py                            39      5    87%
source_s3/source_files_abstract/stream.py                           242     40    83%
source_s3/source_files_abstract/formats/csv_parser.py               102     20    80%
source_s3/s3_utils.py                                                20      4    80%
source_s3/utils.py                                                   31     10    68%
-------------------------------------------------------------------------------------
TOTAL                                                               886     97    89%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [6] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
=========== 105 passed, 6 skipped, 31 warnings in 265.15s (0:04:25) ============

Artem Inzhyyants (artem1205) · 2023-01-10T18:41:28Z

/publish connector=connectors/source-s3

🕑 Publishing the following connectors:
connectors/source-s3
https://github.com/airbytehq/airbyte/actions/runs/3886275941

Connector	Did it publish?	Were definitions generated?
connectors/source-s3	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

* Source S3: update block size for json * Source S3: update docs * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

Source S3: update block size for json

caf4c39

Octavia Squidington IV (octavia-squidington-iv) added area/connectors Connector related issues connectors/source/s3 labels Jan 10, 2023

Source S3: update docs

b93c8f1

Octavia Squidington IV (octavia-squidington-iv) added the area/documentation Improvements or additions to documentation label Jan 10, 2023

Artem Inzhyyants (artem1205) requested review from Arsen Losenko (arsenlosenko) and Baz (bazarnov) January 10, 2023 18:13

Baz (bazarnov) approved these changes Jan 10, 2023

View reviewed changes

auto-bump connector version

2bd5353

Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets January 10, 2023 19:17 — with GitHub Actions Inactive

Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets January 10, 2023 19:18 — with GitHub Actions Inactive

Artem Inzhyyants (artem1205) enabled auto-merge (squash) January 10, 2023 19:24

Artem Inzhyyants (artem1205) merged commit 31edbd8 into master Jan 10, 2023

Artem Inzhyyants (artem1205) deleted the artem1205/source-s3-OC-1257 branch January 10, 2023 19:53

Octavia Squidington III (octavia-squidington-iii) mentioned this pull request Jan 12, 2023

Bump Airbyte version from 0.40.27 to 0.40.28 #21359

Merged

Navigation Menu

Uh oh!

Source S3: update block size for json - #21210

Source S3: update block size for json#21210
Artem Inzhyyants (artem1205) merged 3 commits into
masterfrom
artem1205/source-s3-OC-1257

Artem Inzhyyants (artem1205) commented Jan 10, 2023 •

edited

Loading

Uh oh!

Artem Inzhyyants (artem1205) commented Jan 10, 2023 •

edited by github-actions Bot

Loading

Uh oh!

Artem Inzhyyants (artem1205) commented Jan 10, 2023 •

edited by github-actions Bot

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Artem Inzhyyants (artem1205) commented Jan 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Uh oh!

Artem Inzhyyants (artem1205) commented Jan 10, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

Artem Inzhyyants (artem1205) commented Jan 10, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Artem Inzhyyants (artem1205) commented Jan 10, 2023 •

edited

Loading

Artem Inzhyyants (artem1205) commented Jan 10, 2023 •

edited by github-actions Bot

Loading

Artem Inzhyyants (artem1205) commented Jan 10, 2023 •

edited by github-actions Bot

Loading