feat: Support installing poetry dependencies with pip #311

pdecat · 2022-05-17T10:02:10Z

Description

This PR adds support for installing poetry dependencies with pip --no-deps for projects with pyproject.toml and optionally poetry.lock files.

We use pip as as poetry does not currently provide a proper way to isolate installed dependencies from virtualenv (pip's --target option). --no-deps disables pip's dependency resolver as poetry already pinned all dependencies.

This is a continuation of #186

Motivation and Context

poetry's is more and more used and has a robust dependency resolver.

Breaking Changes

New poetry install step is not enabled by default and requires explicitly setting poetry_install = True in the source block.

How Has This Been Tested?

I have updated at least one of the examples/* to demonstrate and validate my change(s)
I have tested and validated these changes using one or more of the provided examples/* projects

I have executed pre-commit run -a on my pull request

cd examples/build-package
terraform init
rm -rf builds/lambda_layer/ && terraform apply -target module.lambda_layer -auto-approve
rm -rf builds/lambda_layer_poetry/ && terraform apply -target module.lambda_layer_poetry -auto-approve
rm -rf builds/package_dir/ && terraform apply -target module.package_dir -auto-approve
rm -rf builds/package_dir_poetry/ && terraform apply -target module.package_dir_poetry -auto-approve
terraform destroy

pdecat · 2022-05-17T10:10:34Z

examples/build-package/main.tf

@@ -278,8 +302,31 @@ module "lambda_layer" {

  build_in_docker = true
  runtime         = "python3.8"
-  docker_image    = "public.ecr.aws/sam/build-python3.8"
+  docker_image    = "build-python3.8"


If public.ecr.aws/sam/build-python3.8 already exists locally, the docker_file is not used.

pdecat · 2022-05-17T10:15:00Z

package.py

                    pip_requirements_step(
                        os.path.join(path, 'requirements.txt'))


pip_requirements_step() already appends the filename if path is a directory:

Suggested change

pip_requirements_step(

os.path.join(path, 'requirements.txt'))

pip_requirements_step(path)

examples/build-package/main.tf

github-actions · 2022-07-18T00:17:43Z

This PR has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this PR will be closed in 10 days

pdecat · 2022-07-18T08:03:33Z

Still current

github-actions · 2022-08-19T00:20:22Z

This PR has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this PR will be closed in 10 days

pdecat · 2022-08-23T08:47:06Z

Still current

1davidmichael · 2022-09-23T14:01:41Z

I primarily use poetry with Python and appreciate this PR. I hope it is merged at some point in the future. Thanks for this work!

antonbabenko · 2022-09-23T14:06:07Z

@pdecat Please let us know if/when this PR is ready for review and merge?

I expect that other users (including @1davidmichael @panessa @Tonkonozhenko) will be able to help with the review and testing.

pdecat · 2022-09-23T14:09:35Z

Hi, I believe it is ready for review and testing by others.

panessa

I added support for http-basic auth for poetry. Code was not taking in consideration poetry.toml file

package.py

examples/build-package/main.tf

pdecat

Hi @panessa, thanks for the review! I've integrated your suggested changes, please take a look.

examples/build-package/main.tf

pdecat · 2022-09-29T13:41:50Z

Hi @antonbabenko, some Github Actions jobs failed because of network errors. Mind to restart them? 🙏

antonbabenko · 2022-09-29T19:40:32Z

@pdecat Restarted GitHub Actions, and all checks are green. I have scheduled to revive this PR on the 13th of October (after HashiConf). Hopefully, everyone else will be addressing all the comments and do a review, too.

antonbabenko

I just tried to test it locally after #362 was merged, and apparently, it broke package.py you've created.

The error message I got during terraform destroy after the successful run of rm -rf builds/lambda_layer/ && terraform apply -target module.lambda_layer:

│     npm_requirements_step(
│   File ".../terraform-aws-modules/terraform-aws-lambda/examples/build-package/../../package.py", line 709, in npm_requirements_step
│     raise RuntimeError(
│ RuntimeError: Nodejs interpreter version equal to
│ defined lambda runtime (nodejs14.x) should be available
│ in system PATH
│ 
│ State: exit status 1

Could you please fix that and update README.md in the root to showcase poetry support in this section and maybe also in other places where it makes sense?

antonbabenko · 2022-10-21T18:04:28Z

test_package_toml.py

@@ -0,0 +1,10 @@
+from package import get_build_system_from_pyproject_toml


Can we move this file from the root to keep it clean?

Also, could you please update README with some instructions on how to run these? Or just delete it :)

Moved python unit tests to tests/ folder, add instructions to README.

And also added Github Action to run these tests against all supported python versions.

antonbabenko · 2022-10-21T18:07:31Z

examples/fixtures/unittests/pyproject-unknown.toml

@@ -0,0 +1,2 @@
+[build-system]


Do we need this file, too?

Yes, it is needed by python unit tests. Moved it too.

examples/build-package/main.tf

pdecat · 2022-10-22T17:32:55Z

I just tried to test it locally after terraform-linters/tflint#362 was merged, and apparently, it broke package.py you've created.

The error message I got during terraform destroy after the successful run of rm -rf builds/lambda_layer/ && terraform apply -target module.lambda_layer:
│     npm_requirements_step(
│   File ".../terraform-aws-modules/terraform-aws-lambda/examples/build-package/../../package.py", line 709, in npm_requirements_step
│     raise RuntimeError(
│ RuntimeError: Nodejs interpreter version equal to
│ defined lambda runtime (nodejs14.x) should be available
│ in system PATH
│ 
│ State: exit status 1

The same errors happen using current master branch for all examples that do not have build_in_docker = true for which the configured runtime is not available locally.

I see at least 3 solutions:

install all runtimes used in the examples
update all examples to only use locally available runtimes
remove the runtime checks for the examples

As a work-around for now, only destroy the modules that were applied with resource targeting:

terraform destroy -target module.lambda_layer
terraform destroy -target module.lambda_layer_poetry
terraform destroy -target module.package_dir
terraform destroy -target module.package_dir_poetry

…eses tests, and add instructions to README

pdecat · 2022-10-24T22:23:20Z

The Pre-Commit / Max TF pre-commit check is failing with a weird error:

Terraform validate with tflint...........................................Failed
- hook id: terraform_tflint
- exit code: 1

Command 'tflint --init' successfully done:




TFLint in modules/deploy/:
Failed to check ruleset; Failed to check `terraform_deprecated_index` rule: main.tf:52,8-9: Invalid character; This character is not used within the language., and 57 other diagnostic(s)

See https://github.com/terraform-aws-modules/terraform-aws-lambda/actions/runs/3312065827/jobs/5468288824

And does not only affect jobs from this PR, other PRs are also affected, see https://github.com/terraform-aws-modules/terraform-aws-lambda/actions/runs/3313158688/jobs/5470763712

Edit: this seems to be caused by terraform-linters/tflint-ruleset-terraform#42

pdecat · 2022-10-26T13:56:21Z

Pre-commit hooks should be fixed now: https://github.com/terraform-linters/tflint/releases/tag/v0.42.2

antonbabenko

Works like a charm :) Thank you for your contribution!

## [4.3.0](v4.2.2...v4.3.0) (2022-10-31) ### Features * Support installing poetry dependencies with pip ([#311](#311)) ([398ae5a](398ae5a))

antonbabenko · 2022-10-31T14:35:09Z

This PR is included in version 4.3.0 🎉

Tonkonozhenko · 2022-10-31T14:56:49Z

@pdecat @antonbabenko thank you for the work! we'll use it asap

github-actions · 2022-12-01T02:28:56Z

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

antonbabenko · 2023-04-18T09:45:08Z

This issue has been resolved in version 4.16.0 🎉

This was referenced May 17, 2022

feat: Support installing dependencies with poetry #186

Closed

Python - pip install --no-deps #123

Closed

pdecat force-pushed the support_poetry branch from 60794b4 to 46b3e92 Compare May 17, 2022 10:07

pdecat commented May 17, 2022

View reviewed changes

examples/build-package/main.tf Show resolved Hide resolved

pdecat force-pushed the support_poetry branch from 46b3e92 to 65b5f68 Compare June 7, 2022 13:31

pdecat force-pushed the support_poetry branch from 65b5f68 to 7914fab Compare June 17, 2022 14:31

github-actions bot added the stale label Jul 18, 2022

github-actions bot removed the stale label Jul 19, 2022

github-actions bot added the stale label Aug 19, 2022

pdecat force-pushed the support_poetry branch from 7914fab to fdea8b9 Compare August 19, 2022 00:46

github-actions bot removed the stale label Aug 24, 2022

pdecat force-pushed the support_poetry branch from fdea8b9 to c719460 Compare September 2, 2022 08:45

panessa reviewed Sep 28, 2022

View reviewed changes

pdecat force-pushed the support_poetry branch from 64e76a4 to 37d2249 Compare September 29, 2022 09:53

pdecat commented Sep 29, 2022

View reviewed changes

examples/build-package/main.tf Outdated Show resolved Hide resolved

pdecat force-pushed the support_poetry branch 2 times, most recently from b7bdc36 to bdbbb0f Compare September 29, 2022 12:15

antonbabenko requested changes Oct 21, 2022

View reviewed changes

pdecat force-pushed the support_poetry branch 2 times, most recently from fd8940d to 09d78ca Compare October 22, 2022 17:26

pdecat force-pushed the support_poetry branch from 09d78ca to 472312c Compare October 22, 2022 17:26

antonbabenko mentioned this pull request Oct 22, 2022

Code in examples/build-package is failing after #362 was merged #363

Closed

pdecat added 6 commits October 24, 2022 11:35

feat: Support installing dependencies with poetry

b291194

feat: Export poetry dependencies and install them with pip --no-deps

c5c2c16

Use python 3.9 for poetry example everywhere

9dfd7ea

Take poetry.toml into account if it exists

dc01989

Use poetry 1.2.1 in examples

5183b4c

Move python unit tests to tests/ folder, add Github Actions to run th…

c85fddf

…eses tests, and add instructions to README

pdecat force-pushed the support_poetry branch 3 times, most recently from 36db3c8 to 1f960f6 Compare October 24, 2022 22:20

Update versions used in poetry example

876ffaa

pdecat force-pushed the support_poetry branch from 1f960f6 to 876ffaa Compare October 26, 2022 13:56

Merge branch 'master' into support_poetry

c6b43dc

antonbabenko approved these changes Oct 31, 2022

View reviewed changes

antonbabenko merged commit 398ae5a into terraform-aws-modules:master Oct 31, 2022

antonbabenko pushed a commit that referenced this pull request Oct 31, 2022

chore(release): version 4.3.0 [skip ci]

b36268c

## [4.3.0](v4.2.2...v4.3.0) (2022-10-31) ### Features * Support installing poetry dependencies with pip ([#311](#311)) ([398ae5a](398ae5a))

pdecat deleted the support_poetry branch October 31, 2022 14:47

github-actions bot locked as resolved and limited conversation to collaborators Dec 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Support installing poetry dependencies with pip #311

feat: Support installing poetry dependencies with pip #311

pdecat commented May 17, 2022 •

edited

pdecat May 17, 2022

pdecat May 17, 2022

github-actions bot commented Jul 18, 2022

pdecat commented Jul 18, 2022

github-actions bot commented Aug 19, 2022

pdecat commented Aug 23, 2022

1davidmichael commented Sep 23, 2022

antonbabenko commented Sep 23, 2022

pdecat commented Sep 23, 2022

panessa left a comment

pdecat left a comment

pdecat commented Sep 29, 2022

antonbabenko commented Sep 29, 2022

antonbabenko left a comment

antonbabenko Oct 21, 2022

pdecat Oct 22, 2022

antonbabenko Oct 21, 2022

pdecat Oct 22, 2022

pdecat commented Oct 22, 2022 •

edited

pdecat commented Oct 24, 2022 •

edited

pdecat commented Oct 26, 2022

antonbabenko left a comment

antonbabenko commented Oct 31, 2022

Tonkonozhenko commented Oct 31, 2022

github-actions bot commented Dec 1, 2022

antonbabenko commented Apr 18, 2023

		pip_requirements_step(
		os.path.join(path, 'requirements.txt'))

	pip_requirements_step(
	os.path.join(path, 'requirements.txt'))
	pip_requirements_step(path)

		@@ -0,0 +1,10 @@
		from package import get_build_system_from_pyproject_toml

feat: Support installing poetry dependencies with pip #311

feat: Support installing poetry dependencies with pip #311

Conversation

pdecat commented May 17, 2022 • edited

Description

Motivation and Context

Breaking Changes

How Has This Been Tested?

pdecat May 17, 2022

Choose a reason for hiding this comment

pdecat May 17, 2022

Choose a reason for hiding this comment

github-actions bot commented Jul 18, 2022

pdecat commented Jul 18, 2022

github-actions bot commented Aug 19, 2022

pdecat commented Aug 23, 2022

1davidmichael commented Sep 23, 2022

antonbabenko commented Sep 23, 2022

pdecat commented Sep 23, 2022

panessa left a comment

Choose a reason for hiding this comment

pdecat left a comment

Choose a reason for hiding this comment

pdecat commented Sep 29, 2022

antonbabenko commented Sep 29, 2022

antonbabenko left a comment

Choose a reason for hiding this comment

antonbabenko Oct 21, 2022

Choose a reason for hiding this comment

pdecat Oct 22, 2022

Choose a reason for hiding this comment

antonbabenko Oct 21, 2022

Choose a reason for hiding this comment

pdecat Oct 22, 2022

Choose a reason for hiding this comment

pdecat commented Oct 22, 2022 • edited

pdecat commented Oct 24, 2022 • edited

pdecat commented Oct 26, 2022

antonbabenko left a comment

Choose a reason for hiding this comment

antonbabenko commented Oct 31, 2022

Tonkonozhenko commented Oct 31, 2022

github-actions bot commented Dec 1, 2022

antonbabenko commented Apr 18, 2023

pdecat commented May 17, 2022 •

edited

pdecat commented Oct 22, 2022 •

edited

pdecat commented Oct 24, 2022 •

edited