Skip to content

🎉 S3 source: add support for non-AWS S3 Storage - #6398

Merged
Dmytro (avida) merged 3 commits into
masterfrom
drezchykov/s3-custom-storage
Sep 27, 2021
Merged

🎉 S3 source: add support for non-AWS S3 Storage#6398
Dmytro (avida) merged 3 commits into
masterfrom
drezchykov/s3-custom-storage

Conversation

@avida

@avida Dmytro (avida) commented Sep 23, 2021

Copy link
Copy Markdown
Contributor

What

Resolves #5751

How

Describe the solution

Recommended reading order

  1. x.java
  2. y.python

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

  • Community member? Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • docs/SUMMARY.md
    • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
    • docs/integrations/README.md
    • airbyte-integrations/builds.md
  • PR name follows PR naming conventions
  • Connector added to connector index like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • Credentials added to Github CI. Instructions.
  • /test connector=connectors/<name> command is passing.
  • New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions
  • Connector version bumped like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • Credentials added to Github CI. Instructions.
  • /test connector=connectors/<name> command is passing.
  • New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

  • Issue acceptance criteria met
  • PR name follows PR naming conventions
  • If adding a new generator, add it to the list of scaffold modules being tested
  • The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
  • Documentation which references the generator is updated as needed.

@github-actions github-actions Bot added the area/connectors Connector related issues label Sep 23, 2021
@avida

Dmytro (avida) commented Sep 23, 2021

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1265733052
❌ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1265733052
🐛 https://gradle.com/s/drt65ym2b4sys

@avida
Dmytro (avida) temporarily deployed to more-secrets September 23, 2021 12:10 Inactive
@jrhizor
Jared Rhizor (jrhizor) temporarily deployed to more-secrets September 23, 2021 12:12 Inactive
@avida
Dmytro (avida) force-pushed the drezchykov/s3-custom-storage branch from 52e1910 to 79b4b08 Compare September 23, 2021 12:18
@avida
Dmytro (avida) temporarily deployed to more-secrets September 23, 2021 12:20 Inactive
@jrhizor
Jared Rhizor (jrhizor) temporarily deployed to more-secrets September 23, 2021 12:20 Inactive
@avida
Dmytro (avida) force-pushed the drezchykov/s3-custom-storage branch from 79b4b08 to 583acba Compare September 23, 2021 12:42
@avida
Dmytro (avida) temporarily deployed to more-secrets September 23, 2021 12:44 Inactive
@jrhizor
Jared Rhizor (jrhizor) temporarily deployed to more-secrets September 23, 2021 12:45 Inactive
@avida
Dmytro (avida) force-pushed the drezchykov/s3-custom-storage branch from 583acba to 42858c3 Compare September 23, 2021 12:59
@avida

Dmytro (avida) commented Sep 23, 2021

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1265899047
❌ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1265899047
🐛

@avida
Dmytro (avida) temporarily deployed to more-secrets September 23, 2021 13:01 Inactive
@jrhizor
Jared Rhizor (jrhizor) temporarily deployed to more-secrets September 23, 2021 13:01 Inactive
@avida
Dmytro (avida) force-pushed the drezchykov/s3-custom-storage branch 2 times, most recently from 2511742 to 9bc12bb Compare September 23, 2021 13:45
@avida
Dmytro (avida) temporarily deployed to more-secrets September 23, 2021 13:46 Inactive
@jrhizor
Jared Rhizor (jrhizor) temporarily deployed to more-secrets September 23, 2021 13:48 Inactive
@avida
Dmytro (avida) force-pushed the drezchykov/s3-custom-storage branch from 9bc12bb to 0bdf042 Compare September 23, 2021 13:49
@avida

Dmytro (avida) commented Sep 23, 2021

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1266082125
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1266082125
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              158    109    31%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     25    39%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py      75     11    85%
	 ------------------------------------------------------------------------
	 TOTAL                                                  778    432    44%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            83     59    29%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     19    42%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        660    393    40%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 source_s3/__init__.py                                                 2      0   100%
	 source_s3/s3file.py                                                  49      3    94%
	 source_s3/source.py                                                  23      0   100%
	 source_s3/source_files_abstract/__init__.py                          19      0   100%
	 source_s3/source_files_abstract/formats/abstract_file_parser.py      37      2    95%
	 source_s3/source_files_abstract/formats/csv_parser.py                71     20    72%
	 source_s3/source_files_abstract/formats/csv_spec.py                  14      0   100%
	 source_s3/source_files_abstract/formats/parquet_parser.py            41     28    32%
	 source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
	 source_s3/source_files_abstract/source.py                            40     18    55%
	 source_s3/source_files_abstract/spec.py                              42     22    48%
	 source_s3/source_files_abstract/storagefile.py                       16      0   100%
	 source_s3/source_files_abstract/stream.py                           183     11    94%
	 source_s3/stream.py                                                  43      3    93%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                               589    107    82%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 source_s3/__init__.py                                                 2      0   100%
	 source_s3/s3file.py                                                  49     26    47%
	 source_s3/source.py                                                  23      0   100%
	 source_s3/source_files_abstract/__init__.py                          19     11    42%
	 source_s3/source_files_abstract/formats/abstract_file_parser.py      37      0   100%
	 source_s3/source_files_abstract/formats/csv_parser.py                71     19    73%
	 source_s3/source_files_abstract/formats/csv_spec.py                  14      0   100%
	 source_s3/source_files_abstract/formats/parquet_parser.py            41      0   100%
	 source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
	 source_s3/source_files_abstract/source.py                            40     18    55%
	 source_s3/source_files_abstract/spec.py                              42     22    48%
	 source_s3/source_files_abstract/storagefile.py                       16      3    81%
	 source_s3/source_files_abstract/stream.py                           183     94    49%
	 source_s3/stream.py                                                  43     31    28%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                               589    224    62%

@avida
Dmytro (avida) temporarily deployed to more-secrets September 23, 2021 13:51 Inactive
@jrhizor
Jared Rhizor (jrhizor) temporarily deployed to more-secrets September 23, 2021 13:52 Inactive
@avida
Dmytro (avida) force-pushed the drezchykov/s3-custom-storage branch from 0bdf042 to 695515a Compare September 23, 2021 14:19
@avida
Dmytro (avida) temporarily deployed to more-secrets September 23, 2021 14:23 Inactive
@avida
Dmytro (avida) force-pushed the drezchykov/s3-custom-storage branch from 695515a to ac84381 Compare September 24, 2021 08:41
@github-actions github-actions Bot added the area/documentation Improvements or additions to documentation label Sep 24, 2021
@avida
Dmytro (avida) temporarily deployed to more-secrets September 24, 2021 08:43 Inactive
@avida

Dmytro (avida) commented Sep 24, 2021

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1269120537
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1269120537
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              158    109    31%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     25    39%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py      75     11    85%
	 ------------------------------------------------------------------------
	 TOTAL                                                  778    432    44%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            83     59    29%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     19    42%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        660    393    40%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 source_s3/__init__.py                                                 2      0   100%
	 source_s3/s3file.py                                                  49      3    94%
	 source_s3/source.py                                                  23      0   100%
	 source_s3/source_files_abstract/__init__.py                          21      3    86%
	 source_s3/source_files_abstract/formats/abstract_file_parser.py      37      2    95%
	 source_s3/source_files_abstract/formats/csv_parser.py                71     20    72%
	 source_s3/source_files_abstract/formats/csv_spec.py                  14      0   100%
	 source_s3/source_files_abstract/formats/parquet_parser.py            41     28    32%
	 source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
	 source_s3/source_files_abstract/source.py                            40     18    55%
	 source_s3/source_files_abstract/spec.py                              42     22    48%
	 source_s3/source_files_abstract/storagefile.py                       16      0   100%
	 source_s3/source_files_abstract/stream.py                           183     11    94%
	 source_s3/stream.py                                                  43      3    93%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                               591    110    81%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 source_s3/__init__.py                                                 2      0   100%
	 source_s3/s3file.py                                                  49     26    47%
	 source_s3/source.py                                                  23      0   100%
	 source_s3/source_files_abstract/__init__.py                          21     13    38%
	 source_s3/source_files_abstract/formats/abstract_file_parser.py      37      0   100%
	 source_s3/source_files_abstract/formats/csv_parser.py                71     19    73%
	 source_s3/source_files_abstract/formats/csv_spec.py                  14      0   100%
	 source_s3/source_files_abstract/formats/parquet_parser.py            41      0   100%
	 source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
	 source_s3/source_files_abstract/source.py                            40     18    55%
	 source_s3/source_files_abstract/spec.py                              42     22    48%
	 source_s3/source_files_abstract/storagefile.py                       16      3    81%
	 source_s3/source_files_abstract/stream.py                           183     94    49%
	 source_s3/stream.py                                                  43     31    28%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                               591    226    62%

@jrhizor
Jared Rhizor (jrhizor) temporarily deployed to more-secrets September 24, 2021 08:46 Inactive
@avida Dmytro (avida) changed the title S3 source: add custom endpoint 🎉 S3 source: add custom endpoint Sep 24, 2021
- config_path: "secrets/parquet_config.json"
status: "succeed"
- config_path: "integration_tests/invalid_config.json"
status: "failed"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please leave at least one check with the result "failed"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops accidentally removed it

Comment thread airbyte-integrations/connectors/source-s3/integration_tests/spec.json Outdated
Comment thread docs/integrations/sources/s3.md Outdated

@yevhenii-ldv Yevhenii (yevhenii-ldv) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, small comments

@avida
Dmytro (avida) temporarily deployed to more-secrets September 24, 2021 12:36 Inactive
@Phlair George Claireaux (Phlair) changed the title 🎉 S3 source: add custom endpoint 🎉 S3 source: add support for non-AWS S3 Storage Sep 24, 2021

@Phlair George Claireaux (Phlair) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Very cleanly implemented keeping things generic, like it!

Really just minor changes in my comments, main blocking point being the new methods sitting in source_files_abstract/__init__.py

Comment thread airbyte-integrations/connectors/source-s3/acceptance-test-config.yml Outdated
Comment thread airbyte-integrations/connectors/source-s3/integration_tests/acceptance.py Outdated
Comment thread airbyte-integrations/connectors/source-s3/source_s3/source.py Outdated
Comment thread airbyte-integrations/connectors/source-s3/integration_tests/spec.json Outdated
found_a_file = False
try:
for filepath in self.stream_class.filepath_iterator(logger, config.get("provider")):
for filepath in self.stream_class(**config).filepath_iterator():

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was static so that during this connection check we wouldn't be instantiating an instance of the stream_class and running its init method, because our aim is purely to check connection.

I'm not against this change but curious as to why, I may be missing something.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First I've made it non static cause I was storing endpoint not in "provider" but in separate "server" field and it was pain to pass all the configs first to the filepath_iterator and then to _list_backet static functions instead of just passing self. Than I merged "server" field into "provider" and decided to keep filepath_iterator function as instance method in case we would want to pass additional parameters to those functions in the future.

@avida
Dmytro (avida) temporarily deployed to more-secrets September 27, 2021 08:15 Inactive
@avida

Dmytro (avida) commented Sep 27, 2021

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1277463230
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1277463230
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              158    109    31%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     25    39%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py      75     11    85%
	 ------------------------------------------------------------------------
	 TOTAL                                                  778    432    44%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            83     59    29%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     19    42%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        660    393    40%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 source_s3/__init__.py                                                 2      0   100%
	 source_s3/s3_utils.py                                                20      3    85%
	 source_s3/s3file.py                                                  49      3    94%
	 source_s3/source.py                                                  23      0   100%
	 source_s3/source_files_abstract/__init__.py                           0      0   100%
	 source_s3/source_files_abstract/formats/abstract_file_parser.py      37      2    95%
	 source_s3/source_files_abstract/formats/csv_parser.py                71     20    72%
	 source_s3/source_files_abstract/formats/csv_spec.py                  14      0   100%
	 source_s3/source_files_abstract/formats/parquet_parser.py            41     28    32%
	 source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
	 source_s3/source_files_abstract/source.py                            40     18    55%
	 source_s3/source_files_abstract/spec.py                              42     22    48%
	 source_s3/source_files_abstract/storagefile.py                       16      0   100%
	 source_s3/source_files_abstract/stream.py                           183     11    94%
	 source_s3/stream.py                                                  43      3    93%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                               590    110    81%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 source_s3/__init__.py                                                 2      0   100%
	 source_s3/s3_utils.py                                                20     13    35%
	 source_s3/s3file.py                                                  49     26    47%
	 source_s3/source.py                                                  23      0   100%
	 source_s3/source_files_abstract/__init__.py                           0      0   100%
	 source_s3/source_files_abstract/formats/abstract_file_parser.py      37      0   100%
	 source_s3/source_files_abstract/formats/csv_parser.py                71     19    73%
	 source_s3/source_files_abstract/formats/csv_spec.py                  14      0   100%
	 source_s3/source_files_abstract/formats/parquet_parser.py            41      0   100%
	 source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
	 source_s3/source_files_abstract/source.py                            40     18    55%
	 source_s3/source_files_abstract/spec.py                              42     22    48%
	 source_s3/source_files_abstract/storagefile.py                       16      3    81%
	 source_s3/source_files_abstract/stream.py                           183     94    49%
	 source_s3/stream.py                                                  43     31    28%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                               590    226    62%

@jrhizor
Jared Rhizor (jrhizor) temporarily deployed to more-secrets September 27, 2021 08:37 Inactive
@avida

Dmytro (avida) commented Sep 27, 2021

Copy link
Copy Markdown
Contributor Author

/publish connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1278428023
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/1278428023

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connectors Connector related issues area/documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

S3 source should work for non-AWS S3 storages

4 participants