airbyte ci test to support --extras

[erohmensing]

Our airbyte-ci test command supports groups but not extras. This PR allows configuring extras to install if necessary (like when running the unit tests in the upstack PR), and renames to better distinguish between the groups.

We probably don't need to include dev as an optional group since it's implicitly installed, but I didn't mess with that here, for one because I wanted to test the rename.

For usage see upstack PR.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Mar 27, 2024 4:45am

[erohmensing]

• Airbyte CDK: Test via airbyte CI test poetry packages #36497
• airbyte cdk: simplify datamodel-gen scripts #36530
• airbyte ci test to support --extras #36527 👈
• Update out of date internal lockfiles #36535
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @erohmensing and the rest of your teammates on Graphite

[erohmensing]

This was just a note for me when I was working on the upstack PR

There is some weird stuff around keepign this list up to date and also the output of changes being calculated differently from the --modified flag that we should look into later but I haven't addressed in this stack.

[alafanechere]

LGTM.
Nit stacking remark: I would have first renamed extra_poetry_groups to optional_poetry_groups in a PR and would have introduced the poetry_extras in a different one.

• What's exactly the difference between groups and extras?
• Is there a way to install everything in a single poetry command instead of having packages to specify the same things?

[erohmensing]

@alafanechere

What's exactly the difference between groups and extras?

They are admittedly confusing. Groups are only used to organize dependencies, and the very imporant thing is Dependency groups, other than the implicit main group, must only contain dependencies you need in your development process. Installing them is only possible by using Poetry., i.e. they can only be dev dependencies. I think they're basically like dev-extras. I have no idea why you can't organize production dependencies - seems a bit ridiculous.

To declare a set of dependencies, which add additional functionality to the project during runtime, use [extras](https://python-poetry.org/docs/pyproject/#extras) instead. Extras can be installed by the end user using pip., i.e. they are similar to setup.py extras in that they can contain code that is shipped with the package and can be installed by specifying extras.

Is there a way to install everything in a single poetry command instead of having packages to specify the same things?

Not that I can see. There is no command to install all groups, and installing groups is fundamentally different from installing extras.

I guess I don't know if we really need to support groups at all, since we only use dev groups, and that one is implicitly always downloaded (e.g. poetry install == poetry install --with dev.

[alafanechere]

I guess I don't know if we really need to support groups at all, since we only use dev groups, and that one is implicitly always downloaded (e.g. poetry install == poetry install --with dev.

You are right. I think I originally eagerly implemented this because I did not know that poetry install installed dev too and originally packages had different extras name in setup.py for dev deps: tests / dev etc.

Thanks for the clarification on group vs extras. I think it makes sense from a DX standpoint. You don't have to care too much when you run poetry install manually. And when you install packages at build time you can use a flag to ignore dev deps.

[erohmensing]

Merge activity

• Mar 27, 2:16 PM EDT: @erohmensing started a stack merge that includes this pull request via Graphite.
• Mar 27, 2:16 PM EDT: @erohmensing merged this pull request with Graphite.