The charm snap fails to build charm.

[mbruzek]

Checklist

• Confirmed this is an issue with charm-tools, not charmstore-client
• Provide versions of tools used
• Described the feature or ways to replicate the issue

What version am I running?

charm 2.2.0
charm-tools 2.1.5

I am using: Ubuntu Xenial

Issue/Feature

I snap installed charm and tried to build the Kubernetes layers. I encountered an error during the build involving a read-only file system

I expect/expected the following

The snap should be able to build and handle dependencies without errors.

What I got

build: Composing into /home/mbruzek/workspace/charms
build: Destination charm directory: /home/mbruzek/workspace/charms/builds/easyrsa
build: Processing layer: layer:basic
build: Processing layer: layer:leadership
build: Processing layer: easyrsa
build: Processing interface: tls-certificates
Collecting pip
  Using cached pip-9.0.0-py2.py3-none-any.whl
Requirement already up-to-date: wheel in /snap/charm/7/lib/python3.5/site-packages
Installing collected packages: pip
  Found existing installation: pip 8.1.2
    Uninstalling pip-8.1.2:
Exception:
Traceback (most recent call last):
  File "/snap/charm/7/usr/lib/python3.5/shutil.py", line 538, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/snap/charm/7/lib/python3.5/site-packages/pip-8.1.2.dist-info/DESCRIPTION.rst' -> '/tmp/pip-0mm2fqkq-uninstall/snap/charm/7/lib/python3.5/site-packages/pip-8.1.2.dist-info/DESCRIPTION.rst'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/charm/7/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/snap/charm/7/lib/python3.5/site-packages/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/snap/charm/7/lib/python3.5/site-packages/pip/req/req_set.py", line 736, in install
    requirement.uninstall(auto_confirm=True)
  File "/snap/charm/7/lib/python3.5/site-packages/pip/req/req_install.py", line 742, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/snap/charm/7/lib/python3.5/site-packages/pip/req/req_uninstall.py", line 115, in remove
    renames(path, new_path)
  File "/snap/charm/7/lib/python3.5/site-packages/pip/utils/__init__.py", line 267, in renames
    shutil.move(old, new)
  File "/snap/charm/7/usr/lib/python3.5/shutil.py", line 553, in move
    os.unlink(src)
OSError: [Errno 30] Read-only file system: '/snap/charm/7/lib/python3.5/site-packages/pip-8.1.2.dist-info/DESCRIPTION.rst'

[mbruzek]

It looks like the new pip version is causing this problem. We should still consider a fix in the snap package so new python packages do not generate these errors.

[johnsca]

This is a critical bug, since the snap package is automatically updated.

[marcoceppi]

I'm simply unable to replicate this bug which blocks me from patching it

[mbruzek]

Looking at the error, it appears I have/had an older version of pip installed and attempted to do a charm build where it tried to install a new pip in the snap read-only file system. @marcoceppi what is the version in the snap now? Is it possible to recreate (have an older version of pip upgrade to a new one) on a clean xenial system such as a "ubuntu" charm?

[johnsca]

I'm able to reproduce it. Here is the log with -lDEBUG:

build: Composing into /home/johnsca/juju/envs/rq
build: Destination charm directory: /home/johnsca/juju/envs/rq/builds/easyrsa
build: {
  "INTERFACE_PATH": "/home/johnsca/juju/envs/rq/interfaces", 
  "JUJU_REPOSITORY": "/home/johnsca/juju/envs/rq", 
  "LAYER_PATH": "/home/johnsca/juju/envs/rq/layers", 
  "_charm": ".", 
  "_charm_metadata": {
    "description": "This charm delivers the EasyRSA application and through Juju events creates\na Certificate Authority (CA), server certificates, and client certificates.\n", 
    "maintainer": "Matthew Bruzek <matthew.bruzek@canonical.com>", 
    "name": "easyrsa", 
    "provides": {
      "client": {
        "interface": "tls-certificates"
      }
    }, 
    "resources": {
      "easyrsa": {
        "description": "The release of the EasyRSA software you would like to use to create\ncertificate authority (CA) and other Public Key Infrastructure (PKI). \nThis charm was written using v3.0.1, so earlier versions of EasyRSA may \nnot work. You can find the releases of EasyRSA at \nhttps://github.com/OpenVPN/easy-rsa/releases\n", 
        "filename": "easyrsa.tgz", 
        "type": "file"
      }
    }, 
    "series": [
      "xenial", 
      "trusty"
    ], 
    "summary": "Delivers EasyRSA to create a Certificate Authority (CA).", 
    "tags": [
      "tls", 
      "pki", 
      "ca"
    ]
  }, 
  "_name": "easyrsa", 
  "_top_layer": null, 
  "config": "<charmtools.build.config.BuildConfig object at 0x7f7ae4846b50>", 
  "deps": "/home/johnsca/juju/envs/rq/deps", 
  "description": false, 
  "force": false, 
  "hide_metrics": false, 
  "interface_service": "http://interfaces.juju.solutions", 
  "log_level": "DEBUG", 
  "no_local_layers": false, 
  "output_dir": "/home/johnsca/juju/envs/rq", 
  "repo": "/home/johnsca/juju/envs/rq/builds", 
  "report": false, 
  "series": null, 
  "target_dir": "/home/johnsca/juju/envs/rq/builds/easyrsa"
}
charmtools.fetchers: git clone https://github.com/juju-solutions/layer-basic.git /home/johnsca/juju/envs/rq/deps/layer/tmpUu7R3k: Cloning into '/home/johnsca/juju/envs/rq/deps/layer/tmpUu7R3k'...
warning: templates not found /usr/share/git-core/templates

charmtools.fetchers: git clone https://git.launchpad.net/layer-leadership /home/johnsca/juju/envs/rq/deps/layer/tmpA1kEgg: Cloning into '/home/johnsca/juju/envs/rq/deps/layer/tmpA1kEgg'...
warning: templates not found /usr/share/git-core/templates

charmtools.fetchers: git clone https://github.com/juju-solutions/interface-tls-certificates /home/johnsca/juju/envs/rq/deps/interface/tmpRFygHt: Cloning into '/home/johnsca/juju/envs/rq/deps/interface/tmpRFygHt'...
warning: templates not found /usr/share/git-core/templates

build: Processing layer: layer:basic
build: Processing layer: layer:leadership
build: Processing layer: easyrsa
build: Processing interface: tls-certificates
charmtools.build.tactics: Copying layer-basic: /home/johnsca/juju/envs/rq/builds/easyrsa/requirements.txt
charmtools.build.tactics: Copying easyrsa: /home/johnsca/juju/envs/rq/builds/easyrsa/README.md
charmtools.build.tactics: Copying layer-basic: /home/johnsca/juju/envs/rq/builds/easyrsa/hooks/update-status
charmtools.build.tactics: Copying layer-basic: /home/johnsca/juju/envs/rq/builds/easyrsa/hooks/leader-settings-changed
charmtools.build.tactics: Copying layer-basic: /home/johnsca/juju/envs/rq/builds/easyrsa/hooks/config-changed
charmtools.build.tactics: Copying layer-basic: /home/johnsca/juju/envs/rq/builds/easyrsa/hooks/upgrade-charm
charmtools.build.tactics: Copying layer-basic: /home/johnsca/juju/envs/rq/builds/easyrsa/hooks/leader-elected
charmtools.build.tactics: Copying layer-basic: /home/johnsca/juju/envs/rq/builds/easyrsa/hooks/start
charmtools.build.tactics: Copying layer-basic: /home/johnsca/juju/envs/rq/builds/easyrsa/hooks/install
charmtools.build.tactics: Copying layer-basic: /home/johnsca/juju/envs/rq/builds/easyrsa/hooks/hook.template
charmtools.build.tactics: Copying layer-basic: /home/johnsca/juju/envs/rq/builds/easyrsa/hooks/stop
charmtools.build.tactics: Copying layer-basic: /home/johnsca/juju/envs/rq/builds/easyrsa/Makefile
charmtools.build.tactics: Copying layer-leadership: /home/johnsca/juju/envs/rq/builds/easyrsa/reactive/__init__.py
charmtools.build.tactics: Copying easyrsa: /home/johnsca/juju/envs/rq/builds/easyrsa/.gitignore
utils: Already using interpreter /snap/charm/7/usr/bin/python3
PYTHONHOME is set.  You *must* activate the virtualenv before using it
Using base prefix '/snap/charm/7/usr'
New python executable in /tmp/tmpkiEaz2/bin/python3
Also creating executable in /tmp/tmpkiEaz2/bin/python
Installing setuptools, pip, wheel...done.

utils: process: virtualenv --python python3 /tmp/tmpkiEaz2 (0)
utils: Collecting pip
  Using cached pip-9.0.1-py2.py3-none-any.whl
Requirement already up-to-date: wheel in /snap/charm/7/lib/python3.5/site-packages
Installing collected packages: pip
  Found existing installation: pip 8.1.2
    Uninstalling pip-8.1.2:
Exception:
Traceback (most recent call last):
  File "/snap/charm/7/usr/lib/python3.5/shutil.py", line 538, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/snap/charm/7/lib/python3.5/site-packages/pip-8.1.2.dist-info/DESCRIPTION.rst' -> '/tmp/pip-duzfnko7-uninstall/snap/charm/7/lib/python3.5/site-packages/pip-8.1.2.dist-info/DESCRIPTION.rst'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/charm/7/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/snap/charm/7/lib/python3.5/site-packages/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/snap/charm/7/lib/python3.5/site-packages/pip/req/req_set.py", line 736, in install
    requirement.uninstall(auto_confirm=True)
  File "/snap/charm/7/lib/python3.5/site-packages/pip/req/req_install.py", line 742, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/snap/charm/7/lib/python3.5/site-packages/pip/req/req_uninstall.py", line 115, in remove
    renames(path, new_path)
  File "/snap/charm/7/lib/python3.5/site-packages/pip/utils/__init__.py", line 267, in renames
    shutil.move(old, new)
  File "/snap/charm/7/usr/lib/python3.5/shutil.py", line 553, in move
    os.unlink(src)
OSError: [Errno 30] Read-only file system: '/snap/charm/7/lib/python3.5/site-packages/pip-8.1.2.dist-info/DESCRIPTION.rst'
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

utils: process: /tmp/tmpkiEaz2/bin/pip3 install -U pip wheel (2)
Collecting pip
  Using cached pip-9.0.1-py2.py3-none-any.whl
Requirement already up-to-date: wheel in /snap/charm/7/lib/python3.5/site-packages
Installing collected packages: pip
  Found existing installation: pip 8.1.2
    Uninstalling pip-8.1.2:
Exception:
Traceback (most recent call last):
  File "/snap/charm/7/usr/lib/python3.5/shutil.py", line 538, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/snap/charm/7/lib/python3.5/site-packages/pip-8.1.2.dist-info/DESCRIPTION.rst' -> '/tmp/pip-duzfnko7-uninstall/snap/charm/7/lib/python3.5/site-packages/pip-8.1.2.dist-info/DESCRIPTION.rst'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/charm/7/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/snap/charm/7/lib/python3.5/site-packages/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/snap/charm/7/lib/python3.5/site-packages/pip/req/req_set.py", line 736, in install
    requirement.uninstall(auto_confirm=True)
  File "/snap/charm/7/lib/python3.5/site-packages/pip/req/req_install.py", line 742, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/snap/charm/7/lib/python3.5/site-packages/pip/req/req_uninstall.py", line 115, in remove
    renames(path, new_path)
  File "/snap/charm/7/lib/python3.5/site-packages/pip/utils/__init__.py", line 267, in renames
    shutil.move(old, new)
  File "/snap/charm/7/usr/lib/python3.5/shutil.py", line 553, in move
    os.unlink(src)
OSError: [Errno 30] Read-only file system: '/snap/charm/7/lib/python3.5/site-packages/pip-8.1.2.dist-info/DESCRIPTION.rst'
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Command failed: /tmp/tmpkiEaz2/bin/pip3 install -U pip wheel

The issue seems to be that 0637e546/tactics.py#L775 is still trying to upgrade the system pip instead of the venv's pip.

[johnsca]

Note: the log in the above is a bit misleading, in that the utils: process: log line appears after the logged output from the command. So the first error is actually associated with the /tmp/tmpkiEaz2/bin/pip3 install -U pip wheel command, while the second set of error dumps is just that same error being re-raised as fatal.

[marcoceppi]

Would it help if I just re-rolled the snap with the latest version of pip while a fix is worked up so that it doesn't do a system wide pip upgrade?

[marcoceppi]

This is in revision 8 of the charm snap, in the stable channel