Source File: Fix OOM; read Excel files in chunks

[artem1205]

What

Resolving https://github.com/airbytehq/oncall/issues/1871

How

read Excel files in chunks

Recommended reading order

2. y.python

🚨 User Impact 🚨

no breaking changes

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)

• Secrets in the connector's spec are annotated with airbyte_secret

• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.

• Code reviews completed

• Connector version has been incremented

  • Version has been bumped according to our Semantic Versioning for Connectors guidelines
  • Dockerfile has updated version

• Documentation updated

  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example

• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.
• /test connector=connectors/<name> command is passing
• New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

[grubberr]

empty generator can kill you by StopIteration, is it ok ?

[artem1205]

updated;

[artem1205]

/test connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/4817879964
❌ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/4817879964
🐛 https://gradle.com/s/ewcmcrwbzf26a

Build Failed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: Incremental syncs are not supported on this connector.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
======================== 35 passed, 3 skipped in 46.45s ========================
	 =========================== short test summary info ============================
	 FAILED integration_tests/file_formats_test.py::test_local_file_read[excel-xls-8-50-demo]
	 FAILED integration_tests/file_formats_test.py::test_raises_file_wrong_format[csv-csv-excel-demo0]
	 FAILED integration_tests/file_formats_test.py::test_raises_file_wrong_format[csv-csv-excel-demo1]
	 FAILED integration_tests/file_formats_test.py::test_raises_file_wrong_format[jsonl-jsonl-excel-jsonl_nested]
	 �[31m================== �[31m�[1m4 failed�[0m, �[32m36 passed�[0m, �[33m2 warnings�[0m�[31m in 23.93s�[0m�[31m ===================�[0m

[artem1205]

/test connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/4829213026
❌ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/4829213026
🐛

[artem1205]

/test connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/4844572957
✅ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/4844572957
Python tests coverage:

Name                      Stmts   Miss  Cover
---------------------------------------------
source_file/__init__.py       2      0   100%
source_file/utils.py         13      1    92%
source_file/source.py        81      7    91%
source_file/client.py       317     58    82%
---------------------------------------------
TOTAL                       413     66    84%
Name                      Stmts   Miss  Cover
---------------------------------------------
source_file/__init__.py       2      0   100%
source_file/client.py       317     48    85%
source_file/utils.py         13      8    38%
source_file/source.py        81     60    26%
---------------------------------------------
TOTAL                       413    116    72%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: Incremental syncs are not supported on this connector.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
======================== 35 passed, 3 skipped in 51.30s ========================

[artem1205]

/publish connector=connectors/source-file

🕑 Publishing the following connectors:
connectors/source-file
https://github.com/airbytehq/airbyte/actions/runs/4850979954

Connector	Version	Did it publish?	Were definitions generated?
connectors/source-file	0.3.1	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️