Source File: Add Python 3.11 support

[aaronsteers]

Resolves: airbytehq/PyAirbyte#95

What

• Connector cannot be installed on Py3.11, due to missing wheels for the pyarrow library version that we are using.

How

• Bump PyArrow to a more recent version which is compatible with Py3.11.

How to test

%pip install airbyte

import airbyte as ab

source = ab.get_source(
    "source-file",
    pip_url="git+https://www.github.com/airbytehq/airbyte.git@aj/source-file/pyarrow-bump#egg=source-file&subdirectory=airbyte-integrations/connectors/source-file",
)

Recommended reading order

n/a

🚨 User Impact 🚨

None expected.

Pre-merge Actions

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

• Community member? Grant edit access to maintainers (instructions)
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Connector version is set to 0.0.1
  • Dockerfile has version 0.0.1
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • docs/integrations/<source or destination>/<name>.md including changelog with an entry for the initial version. See changelog example
  • docs/integrations/README.md

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Unit & integration tests added

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.

Connector Generator

• Issue acceptance criteria met
• PR name follows PR naming conventions
• If adding a new generator, add it to the list of scaffold modules being tested
• The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:generateScaffolds then checking in your changes
• Documentation which references the generator is updated as needed

Updating the Python CDK

Airbyter

Before merging:

• Pull Request description explains what problem it is solving
• Code change is unit tested
• Build and my-py check pass
• Smoke test the change on at least one affected connector
  • On Github: Run this workflow, passing --use-local-cdk --name=source-<connector> as options
  • Locally: airbyte-ci connectors --use-local-cdk --name=source-<connector> test
• PR is reviewed and approved

After merging:

• Publish the CDK
  • The CDK does not follow proper semantic versioning. Choose minor if this the change has significant user impact or is a breaking change. Choose patch otherwise.
  • Write a thoughtful changelog message so we know what was updated.
• Merge the platform PR that was auto-created for updating the Connector Builder's CDK version
  • This step is optional if the change does not affect the connector builder or declarative connectors.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Mar 5, 2024 2:15am

[aaronsteers]

FYI, Looks like my IDE did auto-cleanup on markdown here.

You can use the ⚙️ icon in the PR diff view to suppress whitespace changes, if helpful.

[aaronsteers]

Tested locally with Py3.11 and PyAirbyte.

Here is my log, pasting here to help us repeat in future:

MyMacBook---GGHWM7QWPJ:PyAirbyte $ pyenv versions
  system
* 3.9.17 (set by /Users/ajsteers/.pyenv/version)
* 3.10.12 (set by /Users/ajsteers/.pyenv/version)
  3.11.4
MyMacBook---GGHWM7QWPJ:PyAirbyte $ mkdir temp && cd temp
MyMacBook---GGHWM7QWPJ:temp $ python3.11 -m venv py311-test
MyMacBook---GGHWM7QWPJ:PyAirbyte $ cd py311-test/
MyMacBook---GGHWM7QWPJ:py311-test $ bin/pip install airbyte
... [long log omitted] ...
MyMacBook---GGHWM7QWPJ:py311-test a$ bin/python3.11
Python 3.11.7 (main, Dec  4 2023, 18:10:11) [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import airbyte as ab
>>> source = ab.get_source("source-file", pip_url="git+https://www.github.com/airbytehq/airbyte.git@aj/source-file/pyarrow-bump#egg=source-file&subdirectory=airbyte-integrations/connectors/source-file")
Installing 'source-file' into virtual environment '/Users/ajsteers/Source/PyAirbyte/py311-test/.venv-source-file'.
... [long log omitted] ...
Connector 'source-file' installed successfully!
For more information, see the source-file documentation:
https://docs.airbyte.com/integrations/sources/file#reference

>>> 

Basically the process is:

1. Run pyenv versions to ensure you have python3.11 installed. (Or use some other method.)
2. Make a new temp folder for our tests: mkdir temp && cd temp
3. Make a new venv using Python 3.11: python3.11 -m venv py311-test
4. Move into the venv (optional, but makes subsequent typing easier): cd py311-test/
5. Install PyAirbyte: bin/pip install airbyte
6. Invoke the Python interpreter: bin/python3.11
7. Import PyAirbyte and test the install process: import airbyte as ab; ab.get_source("source-file", pip_url="git+https://www.github.com/airbytehq/airbyte.git@aj/source-file/pyarrow-bump#egg=source-file&subdirectory=airbyte-integrations/connectors/source-file")