Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace ``python-apt`` functionality #341

Merged
merged 8 commits into from Aug 13, 2019

Conversation

@fnordahl
Copy link
Member

commented Jun 26, 2019

The python-apt package is a wrapper around the apt C++
library which is tightly connected to the version of the
distribution it is shipped on.

This in turn makes it incredibly hard to distribute as a wheel for
a charm that supports a large span of distro versions.

We do not want to rely on system installed Python packages but
distribute the direct charm dependencies as part of the charms
wheelhouse.

As the span of distributions we need to support with reactive
charms widens we will run into compability problems with the
current model.

For further reference see discussion in LP: #1824112 and
juju-solutions/layer-basic#135

@fnordahl fnordahl force-pushed the fnordahl:dist-charm-deps branch 2 times, most recently from 98af3ae to 61c5a33 Jun 26, 2019

@fnordahl

This comment has been minimized.

Copy link
Member Author

commented Jun 26, 2019

Functional test results in pre-existing charms with these bits available [0][1].

0: https://review.opendev.org/#/c/655370/
1: https://review.opendev.org/#/c/655394/

@fnordahl fnordahl force-pushed the fnordahl:dist-charm-deps branch from 61c5a33 to 5b5a482 Jun 27, 2019

@fnordahl

This comment has been minimized.

Copy link
Member Author

commented Jun 27, 2019

Fresh set of functional tests run with these bits on two reactive charms [0][1] and one classic charm [2] in sum spanning trusty through disco.

0: https://review.opendev.org/#/c/655370/
1: https://review.opendev.org/#/c/655394/
2: https://review.opendev.org/#/c/667673/

@ajkavanagh
Copy link
Contributor

left a comment

Looks good; mostly just style, but one is about intent. Please see inline comments.

charmhelpers/fetch/ubuntu.py Outdated Show resolved Hide resolved
charmhelpers/fetch/ubuntu.py Outdated Show resolved Hide resolved
charmhelpers/fetch/ubuntu.py Outdated Show resolved Hide resolved
charmhelpers/fetch/ubuntu.py Outdated Show resolved Hide resolved
charmhelpers/fetch/ubuntu.py Outdated Show resolved Hide resolved
charmhelpers/fetch/ubuntu.py Outdated Show resolved Hide resolved
charmhelpers/fetch/ubuntu.py Outdated Show resolved Hide resolved
charmhelpers/fetch/ubuntu.py Outdated Show resolved Hide resolved
charmhelpers/fetch/ubuntu.py Outdated Show resolved Hide resolved
charmhelpers/fetch/ubuntu.py Outdated Show resolved Hide resolved
@ajkavanagh

This comment has been minimized.

Copy link
Contributor

commented Jun 27, 2019

A thought occurs to me; one not keep the API the same as apk_pkg so it's a plug in replacement?

@fnordahl

This comment has been minimized.

Copy link
Member Author

commented Jul 23, 2019

Yes, you are absolutely right in that that would be a much safer path given we do not control all of our consumers. A set of commits that implement this is on the way.

@fnordahl fnordahl force-pushed the fnordahl:dist-charm-deps branch from 5b5a482 to 8425374 Jul 23, 2019

@fnordahl fnordahl force-pushed the fnordahl:dist-charm-deps branch from 8425374 to bb3a481 Aug 9, 2019

fnordahl added some commits Jun 26, 2019

Add module with subset of ``apt_pkg`` API
The ``python-apt`` package is a wrapper around the ``apt`` C++
library which is tightly connected to the version of the
distribution it is shipped on.

This in turn makes it incredibly hard to distribute as a wheel for
a charm that supports a large span of distro versions.

We do not want to rely on system installed Python packages but
distribute the direct charm dependencies as part of the charms
wheelhouse.

As the span of distributions we need to support with reactive
charms widens we will run into compability problems with the
current model.

For further reference see discussion in LP: #1824112 and
juju-solutions/layer-basic#135
Detect consumer use of upstream ``apt_pkg`` module
When our consumer use the upstream ``apt_pkg`` module in
conjunction with the apt_cache helper function, they may expect us
to call ``apt_pkg.init()`` for them.

Detect this situation, log a warning and make the call to
``apt_pkg.init()`` to avoid the consumer Python interpreter from
crashing with a segmentation fault.

@fnordahl fnordahl force-pushed the fnordahl:dist-charm-deps branch from bb3a481 to 09bb16d Aug 9, 2019

@ajkavanagh
Copy link
Contributor

left a comment

Yes, I like this. A really nice approach to dealing with the apt_pkg issue.

charmhelpers/fetch/ubuntu.py Outdated Show resolved Hide resolved
charmhelpers/fetch/ubuntu_apt_pkg.py Outdated Show resolved Hide resolved

fnordahl added some commits Aug 13, 2019

Update ``apt_cache`` function signature
Make it more clear that any arguments will be ignored.
@ajkavanagh
Copy link
Contributor

left a comment

LGTM

@johnsca
Copy link
Member

left a comment

Looks great. One small comment about a possible interface compatibility concern, but I don't actually know if apt_cache() was ever intended to be used externally, so if it's a non-issue, I'm +1 on merging as-is.

charmhelpers/fetch/ubuntu.py Outdated Show resolved Hide resolved

@fnordahl fnordahl dismissed stale reviews from johnsca and ajkavanagh via 27de385 Aug 13, 2019

@johnsca johnsca merged commit d2ea1b8 into juju:master Aug 13, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

ChrisMacNaughton added a commit to ChrisMacNaughton/charm-helpers that referenced this pull request Aug 14, 2019

Replace ``python-apt`` functionality (juju#341)
* setup: Add missing ``psutil`` dependency

* Add module with subset of ``apt_pkg`` API

The ``python-apt`` package is a wrapper around the ``apt`` C++
library which is tightly connected to the version of the
distribution it is shipped on.

This in turn makes it incredibly hard to distribute as a wheel for
a charm that supports a large span of distro versions.

We do not want to rely on system installed Python packages but
distribute the direct charm dependencies as part of the charms
wheelhouse.

As the span of distributions we need to support with reactive
charms widens we will run into compability problems with the
current model.

For further reference see discussion in LP: #1824112 and
juju-solutions/layer-basic#135

* Make c-h use provided apt_pkg compability shim

* Detect consumer use of upstream ``apt_pkg`` module

When our consumer use the upstream ``apt_pkg`` module in
conjunction with the apt_cache helper function, they may expect us
to call ``apt_pkg.init()`` for them.

Detect this situation, log a warning and make the call to
``apt_pkg.init()`` to avoid the consumer Python interpreter from
crashing with a segmentation fault.

* Improve string handling in ``upstream_version``

* Update ``apt_cache`` function signature

Make it more clear that any arguments will be ignored.

* Use AttributeDict trick for ``_container`` class

* Accept any kw args to apt_cache for compability

openstack-gerrit pushed a commit to openstack/charm-octavia that referenced this pull request Aug 15, 2019

Distribute direct charm dependencies
Do not rely on system installed Python packages but distribute the
direct charm dependencies as part of the charms wheelhouse.

As the span of distributions we need to support with reactive
charms widens we will run into compability problems with the
current model.

For further reference see juju/charm-helpers#341 and
juju-solutions/layer-basic#135

Change-Id: Iacf663d00c5835907cb7ce73c7a97da597780590

openstack-gerrit added a commit to openstack/openstack that referenced this pull request Aug 15, 2019

Update git submodules
* Update charm-octavia from branch 'master'
  - Distribute direct charm dependencies
    
    Do not rely on system installed Python packages but distribute the
    direct charm dependencies as part of the charms wheelhouse.
    
    As the span of distributions we need to support with reactive
    charms widens we will run into compability problems with the
    current model.
    
    For further reference see juju/charm-helpers#341 and
    juju-solutions/layer-basic#135
    
    Change-Id: Iacf663d00c5835907cb7ce73c7a97da597780590

openstack-gerrit added a commit to openstack/openstack that referenced this pull request Aug 15, 2019

Update git submodules
* Update charm-barbican-softhsm from branch 'master'
  - Distribute direct charm dependencies
    
    Do not rely on system installed Python packages but distribute the
    direct charm dependencies as part of the charms wheelhouse.
    
    As the span of distributions we need to support with reactive
    charms widens we will run into compability problems with the
    current model.
    
    For further reference see juju/charm-helpers#341 and
    juju-solutions/layer-basic#135
    
    Change-Id: Ic4e33d59039421593c8ca22f08834ba6f6561b26

openstack-gerrit pushed a commit to openstack/charm-gnocchi that referenced this pull request Aug 15, 2019

Distribute direct charm dependencies
Do not rely on system installed Python packages but distribute the
direct charm dependencies as part of the charms wheelhouse.

As the span of distributions we need to support with reactive
charms widens we will run into compability problems with the
current model.

For further reference see juju/charm-helpers#341 and
juju-solutions/layer-basic#135

Change-Id: I983cb0cb39ae622103e9ff43f89950888e6e033a

openstack-gerrit added a commit to openstack/openstack that referenced this pull request Aug 15, 2019

Update git submodules
* Update charm-gnocchi from branch 'master'
  - Distribute direct charm dependencies
    
    Do not rely on system installed Python packages but distribute the
    direct charm dependencies as part of the charms wheelhouse.
    
    As the span of distributions we need to support with reactive
    charms widens we will run into compability problems with the
    current model.
    
    For further reference see juju/charm-helpers#341 and
    juju-solutions/layer-basic#135
    
    Change-Id: I983cb0cb39ae622103e9ff43f89950888e6e033a

openstack-gerrit pushed a commit to openstack/charm-neutron-dynamic-routing that referenced this pull request Aug 15, 2019

Distribute direct charm dependencies
Do not rely on system installed Python packages but distribute the
direct charm dependencies as part of the charms wheelhouse.

As the span of distributions we need to support with reactive
charms widens we will run into compability problems with the
current model.

For further reference see juju/charm-helpers#341 and
juju-solutions/layer-basic#135

Change-Id: I87c8649d83dd19eeaea17f2a209993cece723983

openstack-gerrit added a commit to openstack/openstack that referenced this pull request Aug 15, 2019

Update git submodules
* Update charm-neutron-dynamic-routing from branch 'master'
  - Distribute direct charm dependencies
    
    Do not rely on system installed Python packages but distribute the
    direct charm dependencies as part of the charms wheelhouse.
    
    As the span of distributions we need to support with reactive
    charms widens we will run into compability problems with the
    current model.
    
    For further reference see juju/charm-helpers#341 and
    juju-solutions/layer-basic#135
    
    Change-Id: I87c8649d83dd19eeaea17f2a209993cece723983

openstack-gerrit pushed a commit to openstack/charm-ceph-rbd-mirror that referenced this pull request Aug 15, 2019

Distribute direct charm dependencies
Do not rely on system installed Python packages but distribute the
direct charm dependencies as part of the charms wheelhouse.

As the span of distributions we need to support with reactive
charms widens we will run into compability problems with the
current model.

For further reference see juju/charm-helpers#341 and
juju-solutions/layer-basic#135

Change-Id: I129ca4506e44abc1e062d5545ff52f992f86a3b9

openstack-gerrit added a commit to openstack/openstack that referenced this pull request Aug 15, 2019

Update git submodules
* Update charm-ceph-rbd-mirror from branch 'master'
  - Distribute direct charm dependencies
    
    Do not rely on system installed Python packages but distribute the
    direct charm dependencies as part of the charms wheelhouse.
    
    As the span of distributions we need to support with reactive
    charms widens we will run into compability problems with the
    current model.
    
    For further reference see juju/charm-helpers#341 and
    juju-solutions/layer-basic#135
    
    Change-Id: I129ca4506e44abc1e062d5545ff52f992f86a3b9

openstack-gerrit pushed a commit to openstack/charm-barbican that referenced this pull request Aug 15, 2019

Distribute direct charm dependencies
Do not rely on system installed Python packages but distribute the
direct charm dependencies as part of the charms wheelhouse.

As the span of distributions we need to support with reactive
charms widens we will run into compability problems with the
current model.

For further reference see juju/charm-helpers#341 and
juju-solutions/layer-basic#135

Change-Id: I0f136f7c1e0e5509f5f2aa5904f89dbd4a227682

openstack-gerrit added a commit to openstack/openstack that referenced this pull request Aug 15, 2019

Update git submodules
* Update charm-barbican from branch 'master'
  - Distribute direct charm dependencies
    
    Do not rely on system installed Python packages but distribute the
    direct charm dependencies as part of the charms wheelhouse.
    
    As the span of distributions we need to support with reactive
    charms widens we will run into compability problems with the
    current model.
    
    For further reference see juju/charm-helpers#341 and
    juju-solutions/layer-basic#135
    
    Change-Id: I0f136f7c1e0e5509f5f2aa5904f89dbd4a227682

openstack-gerrit pushed a commit to openstack/charm-octavia-diskimage-retrofit that referenced this pull request Aug 15, 2019

Distribute direct charm dependencies
Do not rely on system installed Python packages but distribute the
direct charm dependencies as part of the charms wheelhouse.

As the span of distributions we need to support with reactive
charms widens we will run into compability problems with the
current model.

For further reference see juju/charm-helpers#341 and
juju-solutions/layer-basic#135

Change-Id: Ic89bbc395d970039e5da4fe131b0feab9ed0f0a2

openstack-gerrit pushed a commit to openstack/charm-nova-cell-controller that referenced this pull request Aug 15, 2019

Distribute direct charm dependencies
Do not rely on system installed Python packages but distribute the
direct charm dependencies as part of the charms wheelhouse.

As the span of distributions we need to support with reactive
charms widens we will run into compability problems with the
current model.

For further reference see juju/charm-helpers#341 and
juju-solutions/layer-basic#135

Change-Id: I99aa452221bea06331519c3a6cf3fde18c75c879

openstack-gerrit added a commit to openstack/openstack that referenced this pull request Aug 15, 2019

Update git submodules
* Update charm-nova-cell-controller from branch 'master'
  - Distribute direct charm dependencies
    
    Do not rely on system installed Python packages but distribute the
    direct charm dependencies as part of the charms wheelhouse.
    
    As the span of distributions we need to support with reactive
    charms widens we will run into compability problems with the
    current model.
    
    For further reference see juju/charm-helpers#341 and
    juju-solutions/layer-basic#135
    
    Change-Id: I99aa452221bea06331519c3a6cf3fde18c75c879

openstack-gerrit pushed a commit to openstack/charm-designate that referenced this pull request Aug 15, 2019

Distribute direct charm dependencies
Do not rely on system installed Python packages but distribute the
direct charm dependencies as part of the charms wheelhouse.

As the span of distributions we need to support with reactive
charms widens we will run into compability problems with the
current model.

For further reference see juju/charm-helpers#341 and
juju-solutions/layer-basic#135

Change-Id: I03b866648183124cca354c63a81628fcb61ffcd3

openstack-gerrit added a commit to openstack/openstack that referenced this pull request Aug 15, 2019

Update git submodules
* Update charm-designate from branch 'master'
  - Distribute direct charm dependencies
    
    Do not rely on system installed Python packages but distribute the
    direct charm dependencies as part of the charms wheelhouse.
    
    As the span of distributions we need to support with reactive
    charms widens we will run into compability problems with the
    current model.
    
    For further reference see juju/charm-helpers#341 and
    juju-solutions/layer-basic#135
    
    Change-Id: I03b866648183124cca354c63a81628fcb61ffcd3

openstack-gerrit pushed a commit to openstack/charm-pacemaker-remote that referenced this pull request Aug 15, 2019

Distribute direct charm dependencies
Do not rely on system installed Python packages but distribute the
direct charm dependencies as part of the charms wheelhouse.

As the span of distributions we need to support with reactive
charms widens we will run into compability problems with the
current model.

For further reference see juju/charm-helpers#341 and
juju-solutions/layer-basic#135

Change-Id: I23c08f50f1d193a91a02e3f654202a5269ba9590

openstack-gerrit added a commit to openstack/openstack that referenced this pull request Aug 15, 2019

Update git submodules
* Update charm-pacemaker-remote from branch 'master'
  - Distribute direct charm dependencies
    
    Do not rely on system installed Python packages but distribute the
    direct charm dependencies as part of the charms wheelhouse.
    
    As the span of distributions we need to support with reactive
    charms widens we will run into compability problems with the
    current model.
    
    For further reference see juju/charm-helpers#341 and
    juju-solutions/layer-basic#135
    
    Change-Id: I23c08f50f1d193a91a02e3f654202a5269ba9590

openstack-gerrit pushed a commit to openstack/charm-ceph-fs that referenced this pull request Aug 16, 2019

Distribute direct charm dependencies
Do not rely on system installed Python packages but distribute the
direct charm dependencies as part of the charms wheelhouse.

As the span of distributions we need to support with reactive
charms widens we will run into compability problems with the
current model.

For further reference see juju/charm-helpers#341 and
juju-solutions/layer-basic#135

Change-Id: Ie868ed20563d3e37e19ba513ccffe575923bb9e4

openstack-gerrit added a commit to openstack/openstack that referenced this pull request Aug 16, 2019

Update git submodules
* Update charm-ceph-fs from branch 'master'
  - Distribute direct charm dependencies
    
    Do not rely on system installed Python packages but distribute the
    direct charm dependencies as part of the charms wheelhouse.
    
    As the span of distributions we need to support with reactive
    charms widens we will run into compability problems with the
    current model.
    
    For further reference see juju/charm-helpers#341 and
    juju-solutions/layer-basic#135
    
    Change-Id: Ie868ed20563d3e37e19ba513ccffe575923bb9e4

openstack-gerrit pushed a commit to openstack/charm-barbican-vault that referenced this pull request Aug 16, 2019

Distribute direct charm dependencies
Do not rely on system installed Python packages but distribute the
direct charm dependencies as part of the charms wheelhouse.

As the span of distributions we need to support with reactive
charms widens we will run into compability problems with the
current model.

For further reference see juju/charm-helpers#341 and
juju-solutions/layer-basic#135

Change-Id: I29304997cdbe3987fb0e55d3ea67b20550653704

openstack-gerrit added a commit to openstack/openstack that referenced this pull request Aug 16, 2019

Update git submodules
* Update charm-barbican-vault from branch 'master'
  - Distribute direct charm dependencies
    
    Do not rely on system installed Python packages but distribute the
    direct charm dependencies as part of the charms wheelhouse.
    
    As the span of distributions we need to support with reactive
    charms widens we will run into compability problems with the
    current model.
    
    For further reference see juju/charm-helpers#341 and
    juju-solutions/layer-basic#135
    
    Change-Id: I29304997cdbe3987fb0e55d3ea67b20550653704

openstack-gerrit pushed a commit to openstack/charm-vault that referenced this pull request Aug 16, 2019

Distribute direct charm dependencies
Do not rely on system installed Python packages but distribute the
direct charm dependencies as part of the charms wheelhouse.

As the span of distributions we need to support with reactive
charms widens we will run into compability problems with the
current model.

For further reference see juju/charm-helpers#341 and
juju-solutions/layer-basic#135

Change-Id: Id459b11c110ef0454c665c312824791bdb2f0f38

openstack-gerrit added a commit to openstack/openstack that referenced this pull request Aug 16, 2019

Update git submodules
* Update charm-vault from branch 'master'
  - Distribute direct charm dependencies
    
    Do not rely on system installed Python packages but distribute the
    direct charm dependencies as part of the charms wheelhouse.
    
    As the span of distributions we need to support with reactive
    charms widens we will run into compability problems with the
    current model.
    
    For further reference see juju/charm-helpers#341 and
    juju-solutions/layer-basic#135
    
    Change-Id: Id459b11c110ef0454c665c312824791bdb2f0f38

openstack-gerrit pushed a commit to openstack/charm-manila-generic that referenced this pull request Aug 16, 2019

Distribute direct charm dependencies
Do not rely on system installed Python packages but distribute the
direct charm dependencies as part of the charms wheelhouse.

As the span of distributions we need to support with reactive
charms widens we will run into compability problems with the
current model.

For further reference see juju/charm-helpers#341 and
juju-solutions/layer-basic#135

Change-Id: I2d728072bfca20e612c2d4c9178095a22f83631c

openstack-gerrit added a commit to openstack/openstack that referenced this pull request Aug 16, 2019

Update git submodules
* Update charm-manila-generic from branch 'master'
  - Distribute direct charm dependencies
    
    Do not rely on system installed Python packages but distribute the
    direct charm dependencies as part of the charms wheelhouse.
    
    As the span of distributions we need to support with reactive
    charms widens we will run into compability problems with the
    current model.
    
    For further reference see juju/charm-helpers#341 and
    juju-solutions/layer-basic#135
    
    Change-Id: I2d728072bfca20e612c2d4c9178095a22f83631c

openstack-gerrit pushed a commit to openstack/charm-keystone-saml-mellon that referenced this pull request Aug 16, 2019

Distribute direct charm dependencies
Do not rely on system installed Python packages but distribute the
direct charm dependencies as part of the charms wheelhouse.

As the span of distributions we need to support with reactive
charms widens we will run into compability problems with the
current model.

For further reference see juju/charm-helpers#341 and
juju-solutions/layer-basic#135

Change-Id: I11fb4020dc358c40bf9c1870e94186ba7e97c2db

openstack-gerrit added a commit to openstack/openstack that referenced this pull request Aug 16, 2019

Update git submodules
* Update charm-keystone-saml-mellon from branch 'master'
  - Distribute direct charm dependencies
    
    Do not rely on system installed Python packages but distribute the
    direct charm dependencies as part of the charms wheelhouse.
    
    As the span of distributions we need to support with reactive
    charms widens we will run into compability problems with the
    current model.
    
    For further reference see juju/charm-helpers#341 and
    juju-solutions/layer-basic#135
    
    Change-Id: I11fb4020dc358c40bf9c1870e94186ba7e97c2db

openstack-gerrit pushed a commit to openstack/charm-masakari that referenced this pull request Aug 20, 2019

Distribute direct charm dependencies
Do not rely on system installed Python packages but distribute the
direct charm dependencies as part of the charms wheelhouse.

As the span of distributions we need to support with reactive
charms widens we will run into compability problems with the
current model.

For further reference see juju/charm-helpers#341 and
juju-solutions/layer-basic#135

Update domain name uesd in ``.gitreview``.

Change-Id: Ib8476c72020c0a1fbb2549c3ceeafac7f12f3efe

openstack-gerrit added a commit to openstack/openstack that referenced this pull request Aug 20, 2019

Update git submodules
* Update charm-masakari from branch 'master'
  - Distribute direct charm dependencies
    
    Do not rely on system installed Python packages but distribute the
    direct charm dependencies as part of the charms wheelhouse.
    
    As the span of distributions we need to support with reactive
    charms widens we will run into compability problems with the
    current model.
    
    For further reference see juju/charm-helpers#341 and
    juju-solutions/layer-basic#135
    
    Update domain name uesd in ``.gitreview``.
    
    Change-Id: Ib8476c72020c0a1fbb2549c3ceeafac7f12f3efe

openstack-gerrit pushed a commit to openstack/charm-masakari-monitors that referenced this pull request Aug 20, 2019

Distribute direct charm dependencies
Do not rely on system installed Python packages but distribute the
direct charm dependencies as part of the charms wheelhouse.

As the span of distributions we need to support with reactive
charms widens we will run into compability problems with the
current model.

For further reference see juju/charm-helpers#341 and
juju-solutions/layer-basic#135

Fix charm locations in functional test bundles.

Change-Id: I7ad4b548669989f778c604addb1aa873f958a5f9

openstack-gerrit added a commit to openstack/openstack that referenced this pull request Aug 20, 2019

Update git submodules
* Update charm-masakari-monitors from branch 'master'
  - Distribute direct charm dependencies
    
    Do not rely on system installed Python packages but distribute the
    direct charm dependencies as part of the charms wheelhouse.
    
    As the span of distributions we need to support with reactive
    charms widens we will run into compability problems with the
    current model.
    
    For further reference see juju/charm-helpers#341 and
    juju-solutions/layer-basic#135
    
    Fix charm locations in functional test bundles.
    
    Change-Id: I7ad4b548669989f778c604addb1aa873f958a5f9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.