Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

No control over virtualenv used to build wheels #646

Closed
fnordahl opened this issue Oct 10, 2022 · 3 comments · Fixed by #650
Closed

No control over virtualenv used to build wheels #646

fnordahl opened this issue Oct 10, 2022 · 3 comments · Fixed by #650

Comments

@fnordahl
Copy link
Collaborator

Some python packages may require changes to the virtualenv used to build wheels. There is currently no way of doing that.

One example is Jinja2 when building on Focal (ref pallets/jinja#1496):

charm-tools/.github/workflows/tests.yml

Lines 57 to 66 in 977ec1f

- name: Fixup wheelhouse
run: |
set -euxo pipefail
cat << EOF | tee -a tests/charm-minimal/wheelhouse.txt
# https://github.com/pallets/jinja/issues/1496
#
# We ought to teach charm-tools to seed the virtualenv used to build
# wheels with newer versions of pip and setuptools.
Jinja2<3
EOF

The virtualenv is created by charm-tools here:

charm-tools/charmtools/build/tactics.py

Lines 1241 to 1244 in 977ec1f

utils.Process(
('virtualenv', '--python', 'python3', self._venv),
env=self._get_env()
).exit_on_error()()

@facundobatista
Copy link

facundobatista commented Oct 10, 2022 •
edited

+1 to this issue.

When charmcraft packs a charm using the charm plugin, it builds a virtualenv and then allows the user to install packages there before actually dealing the charm requirements. This way, the user can upgrade setuptools and pip, a step that is needed for later proper installation of Jinja2 (the Focal default versions of setuptools and pip do not handle that correctly).

But in the case of charmcraft packing a charm using the reactive plugin, it will end up calling the charm tool, which is the one responsible of the virtualenv creation and installation of charm requirements. As everything is handled in the same charm process, there is no way charmcraft can get in the middle and update these packages "from outside".

So it would be awesome if the charm tool handles this. Thank you!!

@facundobatista facundobatista mentioned this issue Oct 10, 2022
Closed
fnordahl added a commit to openstack-charmers/charm-layer-ovn that referenced this issue Oct 11, 2022
@fnordahl
Pin Jinja2 to avoid build failure on Focal
7acd772 
As concluded in pallets/jinja#1496 Jinja2 requires a newer version
of setuptools than provided in Ubuntu Focal.

It is currently not possible to tell charm-tools to upgrade
setuptools in the virtualenv used for building of wheels
(juju/charm-tools#646), so the only alternative is to pin Jinja2
to avoid the problematic behavior.
fnordahl added a commit to openstack-charmers/charm-layer-ovn that referenced this issue Oct 11, 2022
@fnordahl
Pin Jinja2 to avoid build failure on Focal
3bc3f4e 
As concluded in pallets/jinja#1496 Jinja2 requires a newer version
of setuptools than provided in Ubuntu Focal.

It is currently not possible to tell charm-tools to upgrade
setuptools in the virtualenv used for building of wheels
(juju/charm-tools#646), so the only alternative is to pin Jinja2
to avoid the problematic behavior.
@kot0dama kot0dama mentioned this issue Oct 12, 2022
Closed
@ajkavanagh ajkavanagh mentioned this issue Oct 12, 2022
Closed
This was referenced Nov 6, 2022
Open
Closed
Open
fnordahl added a commit to fnordahl/charm-tools that referenced this issue Nov 12, 2022
@fnordahl
Add option to upgrade core dpes in build venv
bb7306e 
Some dependencies may require newer versions of core dependencies
in order to successfully execute the `pip download` or `pip wheel`
command. Most notable example is Jinja2 and MarkupSafe when
installed on Focal ref pallets/jinja#1496.

At this stage of the process the input from the charm
wheelhouse.txt is not sufficient.

Add an option to upgrade pip and setuptools in the build venv to
deal with these situations.

Fixes: juju#646

Signed-off-by: Frode Nordahl <frode.nordahl@canonical.com>
fnordahl added a commit to fnordahl/charm-tools that referenced this issue Nov 12, 2022
@fnordahl
Add option to upgrade core deps in build venv
85f1e59 
Some packages may require newer versions of core dependencies for
successful execution of the `pip download` or `pip wheel`
commands. Most notable example is Jinja2 and MarkupSafe when
installed on Focal ref pallets/jinja#1496.

At this stage of the process the input from the charm
wheelhouse.txt is not sufficient.

Add an option to upgrade pip and setuptools in the build venv to
deal with these situations.

Fixes: juju#646
Signed-off-by: Frode Nordahl <frode.nordahl@canonical.com>
@fnordahl fnordahl mentioned this issue Nov 12, 2022
Merged
@lathiat
Copy link

lathiat commented Dec 9, 2022

Does this change fix this:
https://discourse.charmhub.io/t/install-or-update-python-packages-before-packing-a-charm/5158

See also: canonical/charmcraft#551

@fnordahl
Copy link
Collaborator Author

fnordahl commented Dec 9, 2022

Does this change fix this: https://discourse.charmhub.io/t/install-or-update-python-packages-before-packing-a-charm/5158

See also: canonical/charmcraft#551

No, when building with the reactive plugin, charm-tools is in charge of the virtualenv, the fix in charmcraft only affects building with the charm plugin for operator framework charms. See also #646 (comment)

This was referenced Dec 21, 2023
Closed
Closed
This was referenced Jan 3, 2024
Closed
Closed
@sudeephb sudeephb mentioned this issue Jan 23, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add option to upgrade core deps in build venv fnordahl/charm-tools
3 participants
@facundobatista @lathiat @fnordahl