Source File: add validate_and_transform of config #18116

grubberr · 2022-10-18T16:16:49Z

Signed-off-by: Sergey Chvalyuk grubberr@gmail.com

What

Try to fix https://github.com/airbytehq/oncall/issues/745

validate and transform reader_options on early stage
Transform Dropbox URL
Remove load_dataframes on "check" stage 🐛 Source File: add data presence validation for HTTPS for check_connection #17504
because it reads only Pandas files, what about YAML files ?
I still think we don't need to read files on "check" stage it's not possible to check everything on "check" stage
just fail on next stage "discover" if needed

How

Describe the solution

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-10-18T19:21:24Z

/test connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/3276290360
❌ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/3276290360
🐛 https://gradle.com/s/mkmuhmkuwz7bu

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED integration_tests/client_storage_providers_test.py::test__read_from_public_provider[HTTPS-https:/storage.googleapis.com/covid19-open-data/v2/latest/epidemiology.csv-10-,-True]
	 FAILED integration_tests/client_storage_providers_test.py::test__read_from_public_provider[HTTPS-storage.googleapis.com/covid19-open-data/v2/latest/epidemiology.csv-10-,-True]
	 FAILED integration_tests/client_storage_providers_test.py::test__read_from_public_provider[local-injected by tests-10-,-True]
	 FAILED integration_tests/client_storage_providers_test.py::test__read_from_public_provider[GCS-gs:/gcp-public-data-landsat/index.csv.gz-18-,-True0]
	 FAILED integration_tests/client_storage_providers_test.py::test__read_from_public_provider[GCS-gs:/gcp-public-data-landsat/index.csv.gz-18-,-True1]
	 FAILED integration_tests/client_storage_providers_test.py::test__read_from_public_provider[S3-s3:/gdelt-open-data/events/20190914.export.csv-58-\\t-False0]
	 FAILED integration_tests/client_storage_providers_test.py::test__read_from_public_provider[S3-s3:/gdelt-open-data/events/20190914.export.csv-58-\\t-False1]
	 FAILED integration_tests/client_storage_providers_test.py::test__read_from_private_gcs
	 FAILED integration_tests/client_storage_providers_test.py::test__read_from_private_aws
	 FAILED integration_tests/client_storage_providers_test.py::test__read_from_public_azblob
	 FAILED integration_tests/client_storage_providers_test.py::test__read_from_private_azblob_shared_key
	 FAILED integration_tests/client_storage_providers_test.py::test__read_from_private_azblob_sas_token
	 �[31m================== �[31m�[1m12 failed�[0m, �[32m21 passed�[0m, �[33m1 warning�[0m�[31m in 30.38s�[0m�[31m ===================�[0m

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-10-19T06:14:49Z

/test connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/3279207635
❌ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/3279207635
🐛 https://gradle.com/s/rbu4h452paata

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED unit_tests/test_backward_compatibility.py - pydantic.error_wrappers.Va...
	 FAILED unit_tests/test_core.py - pydantic.error_wrappers.ValidationError: 1 v...
	 !!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!
	 
	 Results (1.90s):
	 /actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/source-acceptance-test/.venv/lib/python3.9/site-packages/coverage/control.py:788: CoverageWarning: No data was collected. (no-data-collected)
	   self._warn("No data was collected.", slug="no-data-collected")

> Task :airbyte-integrations:connectors:source-file:installLocalReqs
	 Collecting boto3==1.21.21
	   Downloading boto3-1.21.21-py3-none-any.whl (132 kB)
	 Collecting smart-open[all]==6.0.0
	   Downloading smart_open-6.0.0-py3-none-any.whl (58 kB)
	 Collecting lxml==4.9.1
	   Downloading lxml-4.9.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (7.0 MB)
	 Collecting html5lib==1.1
	   Downloading html5lib-1.1-py2.py3-none-any.whl (112 kB)
	 Collecting beautifulsoup4==4.11.1
	   Downloading beautifulsoup4-4.11.1-py3-none-any.whl (128 kB)
	 Collecting pyarrow==9.0.0
	   Downloading pyarrow-9.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (35.3 MB)
	 Collecting xlrd==2.0.1
	   Downloading xlrd-2.0.1-py2.py3-none-any.whl (96 kB)
	 Collecting openpyxl==3.0.10
	   Downloading openpyxl-3.0.10-py2.py3-none-any.whl (242 kB)
	 Collecting pyxlsb==1.0.9
	   Downloading pyxlsb-1.0.9-py2.py3-none-any.whl (23 kB)
	 Collecting soupsieve>1.2
	   Downloading soupsieve-2.3.2.post1-py3-none-any.whl (37 kB)
	 Collecting botocore<1.25.0,>=1.24.21
	   Downloading botocore-1.24.46-py3-none-any.whl (8.7 MB)
	 Collecting s3transfer<0.6.0,>=0.5.0
	   Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB)
	 Collecting jmespath<2.0.0,>=0.7.1
	   Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
	 Collecting google-auth-oauthlib
	   Downloading google_auth_oauthlib-0.5.3-py2.py3-none-any.whl (19 kB)
	 Collecting google-auth>=1.2
	   Downloading google_auth-2.13.0-py2.py3-none-any.whl (174 kB)
	 Collecting requests
	   Using cached requests-2.28.1-py3-none-any.whl (62 kB)
	 Collecting aiohttp
	   Downloading aiohttp-3.8.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)
	 Collecting decorator>4.1.2
	   Downloading decorator-5.1.1-py3-none-any.whl (9.1 kB)
	 Collecting fsspec==2022.7.1
	   Downloading fsspec-2022.7.1-py3-none-any.whl (141 kB)
	 Collecting google-cloud-core<3.0dev,>=2.3.0
	   Downloading google_cloud_core-2.3.2-py2.py3-none-any.whl (29 kB)
	 Collecting google-resumable-media>=2.3.2
	   Downloading google_resumable_media-2.4.0-py2.py3-none-any.whl (77 kB)
	 Collecting google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5
	   Downloading google_api_core-2.10.2-py3-none-any.whl (115 kB)
	 Collecting six>=1.9
	   Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
	 Collecting webencodings
	   Downloading webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
	 Collecting et-xmlfile
	   Downloading et_xmlfile-1.1.0-py3-none-any.whl (4.7 kB)
	 Collecting python-dateutil>=2.8.1
	   Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
	 Collecting numpy>=1.18.5
	   Downloading numpy-1.23.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
	 Collecting pytz>=2020.1
	   Downloading pytz-2022.5-py2.py3-none-any.whl (500 kB)
	 Collecting pynacl>=1.0.1
	   Downloading PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (856 kB)
	 Collecting bcrypt>=3.1.3
	   Downloading bcrypt-4.0.1-cp36-abi3-manylinux_2_28_x86_64.whl (593 kB)
	 Collecting cryptography>=2.5
	   Using cached cryptography-38.0.1-cp36-abi3-manylinux_2_28_x86_64.whl (4.2 MB)
	 Collecting aiobotocore~=2.3.4
	   Downloading aiobotocore-2.3.4-py3-none-any.whl (64 kB)
	 Collecting azure-storage-blob
	   Downloading azure_storage_blob-12.14.1-py3-none-any.whl (383 kB)
	 Collecting azure-core
	   Downloading azure_core-1.26.0-py3-none-any.whl (178 kB)
	 Collecting azure-common
	   Downloading azure_common-1.1.28-py2.py3-none-any.whl (14 kB)
	 Collecting pydantic~=1.6
	   Using cached pydantic-1.9.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.4 MB)
	 Collecting dataclasses-jsonschema==2.15.1
	   Using cached dataclasses_jsonschema-2.15.1-py3-none-any.whl (17 kB)
	 Collecting pendulum
	   Using cached pendulum-2.1.2-cp39-cp39-manylinux1_x86_64.whl (155 kB)
	 Collecting Jinja2~=3.1.2
	   Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
	 Collecting Deprecated~=1.2
	   Using cached Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB)
	 Collecting backoff
	   Using cached backoff-2.2.1-py3-none-any.whl (15 kB)
	 Collecting vcrpy
	   Using cached vcrpy-4.2.1-py2.py3-none-any.whl (40 kB)
	 Collecting ordered-set<4.2.0,>=4.1.0
	   Using cached ordered_set-4.1.0-py3-none-any.whl (7.6 kB)
	 Collecting websocket-client>=0.32.0
	   Using cached websocket_client-1.4.1-py3-none-any.whl (55 kB)
	 Collecting exceptiongroup>=1.0.0rc8
	   Using cached exceptiongroup-1.0.0rc9-py3-none-any.whl (12 kB)
	 Collecting sortedcontainers<3.0.0,>=2.1.0
	   Using cached sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
	 Requirement already satisfied: attrs>=19.2.0 in ./.venv/lib/python3.9/site-packages (from hypothesis~=6.54.1->source-acceptance-test==0.0.0->-r requirements.txt (line 2)) (22.1.0)
	 Collecting pyrsistent>=0.14.0
	   Using cached pyrsistent-0.18.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (115 kB)
	 Requirement already satisfied: setuptools in ./.venv/lib/python3.9/site-packages (from jsonschema~=3.2.0->source-acceptance-test==0.0.0->-r requirements.txt (line 2)) (65.5.0)
	 Collecting wmctrl
	   Using cached wmctrl-0.4-py3-none-any.whl
	 Collecting fancycompleter>=0.8
	   Using cached fancycompleter-0.9.1-py3-none-any.whl (9.7 kB)
	 Collecting pygments
	   Using cached Pygments-2.13.0-py3-none-any.whl (1.1 MB)
	 Requirement already satisfied: typing-extensions>=3.7.4.3 in ./.venv/lib/python3.9/site-packages (from pydantic~=1.6->source-acceptance-test==0.0.0->-r requirements.txt (line 2)) (4.4.0)
	 Requirement already satisfied: pluggy<1.0,>=0.12 in ./.venv/lib/python3.9/site-packages (from pytest~=6.1->source-acceptance-test==0.0.0->-r requirements.txt (line 2)) (0.13.1)
	 Requirement already satisfied: py>=1.8.2 in ./.venv/lib/python3.9/site-packages (from pytest~=6.1->source-acceptance-test==0.0.0->-r requirements.txt (line 2)) (1.11.0)
	 Requirement already satisfied: packaging in ./.venv/lib/python3.9/site-packages (from pytest~=6.1->source-acceptance-test==0.0.0->-r requirements.txt (line 2)) (21.3)
	 Requirement already satisfied: iniconfig in ./.venv/lib/python3.9/site-packages (from pytest~=6.1->source-acceptance-test==0.0.0->-r requirements.txt (line 2)) (1.1.1)
	 Requirement already satisfied: toml in ./.venv/lib/python3.9/site-packages (from pytest~=6.1->source-acceptance-test==0.0.0->-r requirements.txt (line 2)) (0.10.2)
	 Requirement already satisfied: coverage[toml]>=5.2.1 in ./.venv/lib/python3.9/site-packages (from pytest-cov~=3.0.0->source-acceptance-test==0.0.0->-r requirements.txt (line 2)) (6.3.1)
	 Collecting termcolor>=1.1.0
	   Using cached termcolor-2.0.1-py3-none-any.whl (5.4 kB)
	 Collecting botocore<1.25.0,>=1.24.21
	   Downloading botocore-1.24.21-py3-none-any.whl (8.6 MB)
	 Collecting wrapt>=1.10.10
	   Using cached wrapt-1.14.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (77 kB)
	 Collecting aioitertools>=0.5.1
	   Downloading aioitertools-0.11.0-py3-none-any.whl (23 kB)
	 Collecting multidict<7.0,>=4.5
	   Using cached multidict-6.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB)
	 Collecting async-timeout<5.0,>=4.0.0a3
	   Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
	 Collecting charset-normalizer<3.0,>=2.0
	   Using cached charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
	 Collecting frozenlist>=1.1.1
	   Downloading frozenlist-1.3.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (158 kB)
	 Collecting aiosignal>=1.1.2
	   Downloading aiosignal-1.2.0-py3-none-any.whl (8.2 kB)
	 Collecting yarl<2.0,>=1.0
	   Using cached yarl-1.8.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (264 kB)
	 Collecting urllib3<1.27,>=1.25.4
	   Using cached urllib3-1.26.12-py2.py3-none-any.whl (140 kB)
	 Requirement already satisfied: tomli in ./.venv/lib/python3.9/site-packages (from coverage[toml]>=5.2.1->pytest-cov~=3.0.0->source-acceptance-test==0.0.0->-r requirements.txt (line 2)) (2.0.1)
	 Collecting cffi>=1.12
	   Using cached cffi-1.15.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (441 kB)
	 Collecting pyrepl>=0.8.2
	   Using cached pyrepl-0.9.0-py3-none-any.whl
	 Collecting protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5
	   Downloading protobuf-4.21.8-cp37-abi3-manylinux2014_x86_64.whl (408 kB)
	 Collecting googleapis-common-protos<2.0dev,>=1.56.2
	   Downloading googleapis_common_protos-1.56.4-py2.py3-none-any.whl (211 kB)
	 Collecting pyasn1-modules>=0.2.1
	   Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
	 Collecting rsa<5,>=3.1.4
	   Downloading rsa-4.9-py3-none-any.whl (34 kB)
	 Collecting cachetools<6.0,>=2.0.0
	   Downloading cachetools-5.2.0-py3-none-any.whl (9.3 kB)
	 Collecting google-crc32c<2.0dev,>=1.0
	   Downloading google_crc32c-1.5.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (32 kB)
	 Collecting MarkupSafe>=2.0
	   Using cached MarkupSafe-2.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
	 Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.venv/lib/python3.9/site-packages (from packaging->pytest~=6.1->source-acceptance-test==0.0.0->-r requirements.txt (line 2)) (3.0.9)
	 Collecting idna<4,>=2.5
	   Using cached idna-3.4-py3-none-any.whl (61 kB)
	 Collecting certifi>=2017.4.17
	   Using cached certifi-2022.9.24-py3-none-any.whl (161 kB)
	 Collecting msrest>=0.7.1
	   Downloading msrest-0.7.1-py3-none-any.whl (85 kB)
	 Collecting requests-oauthlib>=0.7.0
	   Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
	 Collecting pytzdata>=2020.1
	   Using cached pytzdata-2020.1-py2.py3-none-any.whl (489 kB)
	 Collecting pycparser
	   Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
	 Collecting isodate>=0.6.0
	   Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB)
	 Collecting pyasn1<0.5.0,>=0.4.6
	   Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
	 Collecting oauthlib>=3.0.0
	   Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB)
	 Building wheels for collected packages: genson
	   Building wheel for genson (setup.py): started
	   Building wheel for genson (setup.py): finished with status 'done'
	   Created wheel for genson: filename=genson-1.2.2-py2.py3-none-any.whl size=21269 sha256=5f2a98ce8b44967e078d3d78c52d685b36229efee5ad77ba97153e764767ae85
	   Stored in directory: /root/.cache/pip/wheels/3c/d7/98/a29f29e3861092e784c4c4994b0b30304c79da1bcbc8284af1
	 Successfully built genson
	 Installing collected packages: pyasn1, urllib3, six, rsa, pyasn1-modules, protobuf, idna, charset-normalizer, certifi, cachetools, requests, python-dateutil, pycparser, oauthlib, multidict, jmespath, googleapis-common-protos, google-auth, frozenlist, yarl, requests-oauthlib, pyrsistent, isodate, google-crc32c, google-api-core, cffi, botocore, azure-core, async-timeout, aiosignal, wrapt, sortedcontainers, s3transfer, PyYAML, pytzdata, pyrepl, msrest, MarkupSafe, jsonschema, google-resumable-media, google-cloud-core, exceptiongroup, cryptography, aioitertools, aiohttp, wmctrl, websocket-client, webencodings, vcrpy, termcolor, soupsieve, smart-open, pytz, pynacl, pygments, pydantic, pendulum, ordered-set, numpy, jsonref, Jinja2, hypothesis, google-cloud-storage, google-auth-oauthlib, fsspec, fancycompleter, et-xmlfile, dpath, Deprecated, decorator, dataclasses-jsonschema, boto3, bcrypt, backoff, azure-storage-blob, azure-common, aiobotocore, xlrd, s3fs, requests-mock, pyxlsb, pytest-timeout, pytest-sugar, pytest-mock, pytest-cov, pyarrow, pprintpp, pdbpp, paramiko, pandas, openpyxl, lxml, inflection, icdiff, hypothesis-jsonschema, html5lib, genson, gcsfs, docker, deepdiff, beautifulsoup4, airbyte-cdk, source-file, source-acceptance-test
	   Running setup.py develop for source-file
	   Running setup.py develop for source-acceptance-test
	 Successfully installed Deprecated-1.2.13 Jinja2-3.1.2 MarkupSafe-2.1.1 PyYAML-5.4.1 aiobotocore-2.3.4 aiohttp-3.8.3 aioitertools-0.11.0 aiosignal-1.2.0 airbyte-cdk-0.1.104 async-timeout-4.0.2 azure-common-1.1.28 azure-core-1.26.0 azure-storage-blob-12.14.1 backoff-2.2.1 bcrypt-4.0.1 beautifulsoup4-4.11.1 boto3-1.21.21 botocore-1.24.21 cachetools-5.2.0 certifi-2022.9.24 cffi-1.15.1 charset-normalizer-2.1.1 cryptography-38.0.1 dataclasses-jsonschema-2.15.1 decorator-5.1.1 deepdiff-5.8.1 docker-5.0.3 dpath-2.0.6 et-xmlfile-1.1.0 exceptiongroup-1.0.0rc9 fancycompleter-0.9.1 frozenlist-1.3.1 fsspec-2022.7.1 gcsfs-2022.7.1 genson-1.2.2 google-api-core-2.10.2 google-auth-2.13.0 google-auth-oauthlib-0.5.3 google-cloud-core-2.3.2 google-cloud-storage-2.5.0 google-crc32c-1.5.0 google-resumable-media-2.4.0 googleapis-common-protos-1.56.4 html5lib-1.1 hypothesis-6.54.6 hypothesis-jsonschema-0.20.1 icdiff-1.9.1 idna-3.4 inflection-0.5.1 isodate-0.6.1 jmespath-1.0.1 jsonref-0.2 jsonschema-3.2.0 lxml-4.9.1 msrest-0.7.1 multidict-6.0.2 numpy-1.23.4 oauthlib-3.2.2 openpyxl-3.0.10 ordered-set-4.1.0 pandas-1.4.3 paramiko-2.11.0 pdbpp-0.10.3 pendulum-2.1.2 pprintpp-0.4.0 protobuf-4.21.8 pyarrow-9.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pydantic-1.9.2 pygments-2.13.0 pynacl-1.5.0 pyrepl-0.9.0 pyrsistent-0.18.1 pytest-cov-3.0.0 pytest-mock-3.6.1 pytest-sugar-0.9.5 pytest-timeout-1.4.2 python-dateutil-2.8.2 pytz-2022.5 pytzdata-2020.1 pyxlsb-1.0.9 requests-2.28.1 requests-mock-1.9.3 requests-oauthlib-1.3.1 rsa-4.9 s3fs-2022.7.1 s3transfer-0.5.2 six-1.16.0 smart-open-6.0.0 sortedcontainers-2.4.0 soupsieve-2.3.2.post1 source-acceptance-test-0.0.0 source-file-0.0.0 termcolor-2.0.1 urllib3-1.26.12 vcrpy-4.2.1 webencodings-0.5.1 websocket-client-1.4.1 wmctrl-0.4 wrapt-1.14.1 xlrd-2.0.1 yarl-1.8.1
	 WARNING: You are using pip version 21.3.1; however, version 22.3 is available.
	 You should consider upgrading via the '/actions-runner/_work/airbyte/airbyte/airbyte-integrations/connectors/source-file/.venv/bin/python -m pip install --upgrade pip' command.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
39 actionable tasks: 26 executed, 13 up-to-date

Publishing build scan...
https://gradle.com/s/rbu4h452paata

grubberr · 2022-10-19T13:40:18Z

/test connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/3282072517
❌ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/3282072517
🐛 https://gradle.com/s/7kdxwedqu5ea4

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED unit_tests/test_source.py::test_nan_to_null - pydantic.error_wrappers....
	 �[31m========================= �[31m�[1m1 failed�[0m, �[32m31 passed�[0m�[31m in 2.63s�[0m�[31m =========================�[0m

grubberr · 2022-10-19T13:41:37Z

/test connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/3282081490
❌ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/3282081490
🐛 https://gradle.com/s/xdi72ypuxhwmi

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED unit_tests/test_source.py::test_nan_to_null - pydantic.error_wrappers....
	 �[31m========================= �[31m�[1m1 failed�[0m, �[32m31 passed�[0m�[31m in 2.60s�[0m�[31m =========================�[0m

grubberr · 2022-10-19T14:50:25Z

/test connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/3282612933
❌ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/3282612933
🐛 https://gradle.com/s/2mq3cby5ljycq

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED unit_tests/test_source.py::test_nan_to_null - pydantic.error_wrappers....
	 �[31m========================= �[31m�[1m1 failed�[0m, �[32m31 passed�[0m�[31m in 2.55s�[0m�[31m =========================�[0m

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-10-19T15:18:20Z

/test connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/3282831736
✅ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/3282831736
Python tests coverage:

Name                      Stmts   Miss  Cover
---------------------------------------------
source_file/__init__.py       2      0   100%
source_file/client.py       273     41    85%
source_file/utils.py          9      4    56%
source_file/source.py        65     33    49%
---------------------------------------------
TOTAL                       349     78    78%
Name                      Stmts   Miss  Cover
---------------------------------------------
source_file/utils.py          9      0   100%
source_file/__init__.py       2      0   100%
source_file/source.py        65      4    94%
source_file/client.py       273     34    88%
---------------------------------------------
TOTAL                       349     38    89%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_incremental.py       145     20    86%   21-23, 29-31, 36-43, 48-61, 224
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     10    87%   15-16, 24-30, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1351    451    67%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config
=================== 26 passed, 1 skipped in 63.66s (0:01:03) ===================

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-10-19T18:12:10Z

/test connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/3283949118
✅ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/3283949118
Python tests coverage:

Name                      Stmts   Miss  Cover
---------------------------------------------
source_file/__init__.py       2      0   100%
source_file/client.py       273     41    85%
source_file/utils.py          9      4    56%
source_file/source.py        65     33    49%
---------------------------------------------
TOTAL                       349     78    78%
Name                      Stmts   Miss  Cover
---------------------------------------------
source_file/utils.py          9      0   100%
source_file/__init__.py       2      0   100%
source_file/source.py        65      4    94%
source_file/client.py       273     34    88%
---------------------------------------------
TOTAL                       349     38    89%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_incremental.py       145     20    86%   21-23, 29-31, 36-43, 48-61, 224
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     10    87%   15-16, 24-30, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1351    451    67%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config
=================== 26 passed, 1 skipped in 61.84s (0:01:01) ===================

grubberr · 2022-10-19T18:26:43Z

/publish connector=connectors/source-file

🕑 Publishing the following connectors:
connectors/source-file
https://github.com/airbytehq/airbyte/actions/runs/3284040175

Connector	Did it publish?	Were definitions generated?
connectors/source-file	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

improve reader_options

a092f77

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

github-actions bot added the area/connectors Connector related issues label Oct 18, 2022

octavia-squidington-iv added the connectors/source/file label Oct 18, 2022

grubberr added 2 commits October 18, 2022 22:16

dropbox_force_download added

b03abdc

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

revert back old check

c4c5d48

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr self-assigned this Oct 18, 2022

file.md updated

b8ff347

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

github-actions bot added the area/documentation Improvements or additions to documentation label Oct 18, 2022

grubberr added 2 commits October 19, 2022 08:41

Merge branch 'master' into grubberr/oncall-745-source-file

1368a82

integration_tests fixed

3a0bb83

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Merge branch 'master' into grubberr/oncall-745-source-file

212167e

Merge branch 'master' into grubberr/oncall-745-source-file

35e97ea

fix 'supported_sync_modes' added

b000583

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr requested review from arsenlosenko and davydov-d October 19, 2022 15:53

davydov-d approved these changes Oct 19, 2022

View reviewed changes

grubberr added 2 commits October 19, 2022 21:06

Merge branch 'master' into grubberr/oncall-745-source-file

bbb8dec

fix

72a45f8

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

auto-bump connector version

dcbb444

octavia-squidington-iii temporarily deployed to more-secrets October 19, 2022 18:50 Inactive

grubberr merged commit 5b81897 into master Oct 19, 2022

grubberr deleted the grubberr/oncall-745-source-file branch October 19, 2022 19:10

grubberr mentioned this pull request Oct 19, 2022

Source file secure 0.2.26 #18180

Merged

octavia-squidington-iii mentioned this pull request Oct 20, 2022

Bump Airbyte version from 0.40.15 to 0.40.16 #18268

Merged

jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022

Source File: add validate_and_transform of config (airbytehq#18116)

22d9b1c

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source File: add validate_and_transform of config #18116

Source File: add validate_and_transform of config #18116

grubberr commented Oct 18, 2022 •

edited

Loading

grubberr commented Oct 18, 2022 •

edited by github-actions bot

Loading

grubberr commented Oct 19, 2022 •

edited by github-actions bot

Loading

grubberr commented Oct 19, 2022 •

edited by github-actions bot

Loading

grubberr commented Oct 19, 2022 •

edited by github-actions bot

Loading

grubberr commented Oct 19, 2022 •

edited by github-actions bot

Loading

grubberr commented Oct 19, 2022 •

edited by github-actions bot

Loading

grubberr commented Oct 19, 2022 •

edited by github-actions bot

Loading

grubberr commented Oct 19, 2022 •

edited by github-actions bot

Loading

Source File: add validate_and_transform of config #18116

Source File: add validate_and_transform of config #18116

Conversation

grubberr commented Oct 18, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

grubberr commented Oct 18, 2022 • edited by github-actions bot Loading

Build Failed

grubberr commented Oct 19, 2022 • edited by github-actions bot Loading

Build Failed

grubberr commented Oct 19, 2022 • edited by github-actions bot Loading

Build Failed

grubberr commented Oct 19, 2022 • edited by github-actions bot Loading

Build Failed

grubberr commented Oct 19, 2022 • edited by github-actions bot Loading

Build Failed

grubberr commented Oct 19, 2022 • edited by github-actions bot Loading

Build Passed

grubberr commented Oct 19, 2022 • edited by github-actions bot Loading

Build Passed

grubberr commented Oct 19, 2022 • edited by github-actions bot Loading

grubberr commented Oct 18, 2022 •

edited

Loading

grubberr commented Oct 18, 2022 •

edited by github-actions bot

Loading

grubberr commented Oct 19, 2022 •

edited by github-actions bot

Loading

grubberr commented Oct 19, 2022 •

edited by github-actions bot

Loading

grubberr commented Oct 19, 2022 •

edited by github-actions bot

Loading

grubberr commented Oct 19, 2022 •

edited by github-actions bot

Loading

grubberr commented Oct 19, 2022 •

edited by github-actions bot

Loading

grubberr commented Oct 19, 2022 •

edited by github-actions bot

Loading

grubberr commented Oct 19, 2022 •

edited by github-actions bot

Loading