lxd: snapcraft refresh in containers #1412

kalikiana · Jul 17, 2017

Only update apt cache and install snapcraft once
Allow snapcraft update to upgrade packages and snaps

See the forum proposal.

Note: The other part is implemented in #1546.

sergiusens · Sep 1, 2017

sergiusens reviewed Sep 1, 2017

View changes

as discussed on the forums, this should use refresh

kyrofa · Sep 8, 2017

kyrofa reviewed Sep 8, 2017

View changes

Looking good, few suggestions though, particularly regarding terminology and where the refresh feature landed.

This isn't part of the lifecycle. In order to prevent this module from becoming a catchall, I suggest you create a new module.

Good forward thinking, there. Helpful message.

This seems like a separate feature/bugfix, no?

Same forum topic, arguably separate fix. I dropped it from this one and proposed #1546.

Again, not a lifecycle step. Please change terminology or refactor this function a bit.

kyrofa · Sep 12, 2017

Whoa, something weird is happening here, any ideas? I didn't even get far enough to test this. Logged LP: #1716738.

$ SNAPCRAFT_CONTAINER_BUILDS=1 snapcraft
"grade" property not specified: defaulting to "stable"
Creating snapcraft-opencv-example
error: Error calling 'lxd forkstart snapcraft-opencv-example /var/lib/lxd/containers /var/log/lxd/snapcraft-opencv-example/lxc.conf': err='Failed to run: /usr/bin/lxd forkstart snapcraft-opencv-example /var/lib/lxd/containers /var/log/lxd/snapcraft-opencv-example/lxc.conf: '
  lxc 20170912155351.183 ERROR lxc_start - start.c:lxc_spawn:1186 - Failed to set up id mapping.
  lxc 20170912155351.244 ERROR lxc_start - start.c:__lxc_start:1358 - Failed to spawn container "snapcraft-opencv-example".
  lxc 20170912155351.805 ERROR lxc_conf - conf.c:run_buffer:416 - Script exited with status 1.
  lxc 20170912155351.805 ERROR lxc_start - start.c:lxc_fini:546 - Failed to run lxc.hook.post-stop for container "snapcraft-opencv-example".
  lxc 20170912155351.809 ERROR lxc_conf - conf.c:userns_exec_1:4608 - Error setting up child mappings
  lxc 20170912155351.809 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1288 - Error destroying /sys/fs/cgroup/systemd//lxc/snapcraft-opencv-example
  lxc 20170912155351.812 ERROR lxc_conf - conf.c:userns_exec_1:4608 - Error setting up child mappings
  lxc 20170912155351.812 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1288 - Error destroying /sys/fs/cgroup/devices//lxc/snapcraft-opencv-example
  lxc 20170912155351.814 ERROR lxc_conf - conf.c:userns_exec_1:4608 - Error setting up child mappings
  lxc 20170912155351.814 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1288 - Error destroying /sys/fs/cgroup/pids//lxc/snapcraft-opencv-example
  lxc 20170912155351.816 ERROR lxc_conf - conf.c:userns_exec_1:4608 - Error setting up child mappings
  lxc 20170912155351.816 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1288 - Error destroying /sys/fs/cgroup/cpu//lxc/snapcraft-opencv-example
  lxc 20170912155351.818 ERROR lxc_conf - conf.c:userns_exec_1:4608 - Error setting up child mappings
  lxc 20170912155351.818 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1288 - Error destroying /sys/fs/cgroup/perf_event//lxc/snapcraft-opencv-example
  lxc 20170912155351.820 ERROR lxc_conf - conf.c:userns_exec_1:4608 - Error setting up child mappings
  lxc 20170912155351.820 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1288 - Error destroying /sys/fs/cgroup/blkio//lxc/snapcraft-opencv-example
  lxc 20170912155351.821 ERROR lxc_conf - conf.c:userns_exec_1:4608 - Error setting up child mappings
  lxc 20170912155351.821 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1288 - Error destroying /sys/fs/cgroup/hugetlb//lxc/snapcraft-opencv-example
  lxc 20170912155351.824 ERROR lxc_conf - conf.c:userns_exec_1:4608 - Error setting up child mappings
  lxc 20170912155351.824 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1288 - Error destroying /sys/fs/cgroup/net_cls//lxc/snapcraft-opencv-example
  lxc 20170912155351.827 ERROR lxc_conf - conf.c:userns_exec_1:4608 - Error setting up child mappings
  lxc 20170912155351.827 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1288 - Error destroying /sys/fs/cgroup/freezer//lxc/snapcraft-opencv-example
  lxc 20170912155351.829 ERROR lxc_conf - conf.c:userns_exec_1:4608 - Error setting up child mappings
  lxc 20170912155351.829 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1288 - Error destroying /sys/fs/cgroup/cpuset//lxc/snapcraft-opencv-example
  lxc 20170912155351.831 ERROR lxc_conf - conf.c:userns_exec_1:4608 - Error setting up child mappings
  lxc 20170912155351.831 ERROR lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1288 - Error destroying /sys/fs/cgroup/memory//lxc/snapcraft-opencv-example

Try `lxc info --show-log local:snapcraft-opencv-example` for more info
Stopping local:snapcraft-opencv-example
error: The container is already stopped
Try `lxc info --show-log local:snapcraft-opencv-example` for more info
Traceback (most recent call last):
  File "/home/kyrofa/src/snapcraft/snapcraft/internal/lxd.py", line 148, in _ensure_started
    self._ensure_container()
  File "/home/kyrofa/src/snapcraft/snapcraft/internal/lxd.py", line 330, in _ensure_container
    'lxc', 'start', self._container_name])
  File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['lxc', 'start', 'local:snapcraft-opencv-example']' returned non-zero exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kyrofa/src/snapcraft/bin/snapcraft", line 36, in <module>
    obj=dict(project=snapcraft.ProjectOptions()))
  File "/usr/lib/python3/dist-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1037, in invoke
    return Command.invoke(self, ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/kyrofa/src/snapcraft/snapcraft/cli/__init__.py", line 112, in run
    ctx.forward(lifecyclecli.commands['snap'])
  File "/usr/lib/python3/dist-packages/click/core.py", line 552, in forward
    return self.invoke(cmd, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/kyrofa/src/snapcraft/snapcraft/cli/lifecycle.py", line 129, in snap
    lifecycle.containerbuild('snap', project_options, output, directory)
  File "/home/kyrofa/src/snapcraft/snapcraft/internal/lifecycle.py", line 314, in containerbuild
    metadata=config.get_metadata()).execute(step, args)
  File "/home/kyrofa/src/snapcraft/snapcraft/internal/lxd.py", line 359, in execute
    super().execute(step, args)
  File "/home/kyrofa/src/snapcraft/snapcraft/internal/lxd.py", line 164, in execute
    with self._container_running():
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/home/kyrofa/src/snapcraft/snapcraft/internal/lxd.py", line 131, in _container_running
    with self._ensure_started():
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/home/kyrofa/src/snapcraft/snapcraft/internal/lxd.py", line 154, in _ensure_started
    check_call(['lxc', 'stop', '-f', self._container_name])
  File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['lxc', 'stop', '-f', 'local:snapcraft-opencv-example']' returned non-zero exit status 1

kyrofa · Sep 12, 2017

kyrofa dismissed their stale review Sep 12, 2017

Changes look good, needs testing now.

kyrofa · Sep 12, 2017

Note that I get the same issues just running lxc start snapcraft-opencv-example. But launching new ones/starting them using lxc works fine.

kalikiana · Oct 4, 2017

kalikiana dismissed sergiusens’s stale review Oct 4, 2017

New logic has been split into a refresh command as discussed

kalikiana · Oct 20, 2017

_ensure_container is apparently too complex now - I'm thinking #1627 should be merged first because it's already simplifying that code path.

kalikiana · Oct 23, 2017

Closing for now - will update and re-open once #1627 is in.

kyrofa · Oct 27, 2017

#1627 is in, now.

sergiusens · Nov 7, 2017

@kalikiana did this have thorough testing on your side?

kalikiana · Nov 7, 2017

I did some random testing of this. For more reliable results a log of specific steps tested just now on integration_tests/snaps/go-cgo (roman numbers mean repeated execution):

Run 'SNAPCRAFT_CONTAINER_BUILDS=1 snapcraft pull' on master.
1. Container created, artifacts present
Run SNAPCRAFT_CONTAINER_BUILDS=1 snapcraft pull and all following steps on this branch.
1. No-op
SNAPCRAFT_CONTAINER_BUILDS=1 snapcraft refresh
1. Packages updated, "All snaps up to date"
2. No packages updated, "All snaps up to date"
SNAPCRAFT_CONTAINER_BUILDS=1 snapcraft pull
1. "already ran"
SNAPCRAFT_CONTAINER_BUILDS=1 snapcraft clean
1. Container deleted, artifacts deleted
2. No-op
SNAPCRAFT_CONTAINER_BUILDS=1 snapcraft pull
1. Container created
2. No-op
SNAPCRAFT_CONTAINER_BUILDS=1 snapcraft refresh
1. Packages updated, "All snaps up to date"
2. No packages updated, "All snaps up to date"
snapcraft cleanbuild
1. Retrieved go-cgo_0.1_amd64.snap

sergiusens · Nov 8, 2017

sergiusens reviewed Nov 8, 2017

View changes

Can you omit SNAPCRAFT_CONTAINER_BUILDS and instead make this command optionally show up by means of doing

if not os.environ.get('SNAPCRAFT_CONTAINER_BUILDS'):
    command.pop(commands.index('refresh'))

Right after https://github.com/snapcore/snapcraft/blob/master/snapcraft/cli/__init__.py#L92

Done. I used use_container here since eventually it won't be just the environment variable.

sergiusens · Nov 8, 2017

sergiusens approved these changes Nov 8, 2017

View changes

thanks

Snapcraft update in containers

3faa40d

kalikiana referenced this pull request Jul 17, 2017
Merged
lxd: Inject snapcraft and core snaps into the container #1364

Merge branch 'master' into containerized_update

f8e4a02

kalikiana self-assigned this Jul 19, 2017

kalikiana requested a review from sergiusens Jul 19, 2017

kalikiana added some commits Jul 27, 2017

Merge branch 'master' into containerized_update

9471210

Fix unit tests

049385b

Merge branch 'master' into containerized_update

51d295e

Fix update unit test

5cf5793

sergiusens added this to the 2.34 milestone Aug 3, 2017

kalikiana changed the title from lxd: Snapcraft update in containers to lxd: snapcraft update in containers Aug 8, 2017

kalikiana added some commits Aug 8, 2017

Merge branch 'master' of https://github.com/snapcore/snapcraft into c…
…ontainerized_update

8733bf4

Update update unit tests

c2e9471

sergiusens referenced this pull request Aug 9, 2017
Closed
update story #1462

Merge branch 'master' into containerized_update

3fbc402

sergiusens changed the title from lxd: snapcraft update in containers to lxd: snapcraft refresh in containers Aug 30, 2017

sergiusens modified the milestones: 2.35, 2.34 Sep 6, 2017

kalikiana added some commits Sep 7, 2017

Merge branch 'master' into containerized_update

1856f41

Split off refresh into a dedicated command

636d53c

kalikiana added some commits Sep 12, 2017

Move refresh command into containers module

56213fd

Refactor execute to allow for refresh method

fabf21f

Drop update command running in container

6ae4ec5

Rename and simplify test_refresh

0ca8911

kyrofa dismissed their stale review Sep 12, 2017

kalikiana added some commits Sep 13, 2017

Merge branch 'master' into containerized_update

0276b6f

Merge branch 'master' into containerized_update

0156c17

Use new get_container_config

9090387

kalikiana dismissed sergiusens’s stale review Oct 4, 2017

Merge branch 'master' into containerized_update

b4d7054

kalikiana referenced this pull request Oct 12, 2017
Merged
cli: update parts cache in the container #1546
6 of 6 tasks complete

Merge branch 'master' into containerized_update

557955d

kalikiana closed this Oct 23, 2017

Merge branch 'master' into containerized_update

1fc9164

kalikiana reopened this Nov 3, 2017

Harden the unit tests to capture wrong network checks

dd83421

sergiusens merged commit 3d511c5 into snapcore:master Nov 9, 2017
1 check passed

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

+        +@lifecyclecli.command()
+        +@click.option('--debug', is_flag=True,
+        +              help='Shells into the environment if the build fails.')
+        +def refresh(debug, **kwargs):

+        +            "SNAPCRAFT_CONTAINER_BUILDS is not set or 0.\n"
+        +            "Maybe you meant to update the parts cache instead? "
+        +            "You can do that with the following command:\n\n"
+        +            "snapcraft update")

+        +        project_options = get_project_options(**kwargs)
+        +        lifecycle.containerbuild('update', project_options)
+        +    else:
+        +        remote_parts.update()

snapcore/snapcraft

lxd: snapcraft refresh in containers #1412

kalikiana commented Jul 17, 2017 •

Edited 1 time

kalikiana
Aug 25, 2017

kalikiana referenced this pull request Jul 17, 2017

lxd: Inject snapcraft and core snaps into the container #1364

kalikiana self-assigned this Jul 19, 2017

kalikiana requested a review from sergiusens Jul 19, 2017

kalikiana added some commits Jul 27, 2017

sergiusens added this to the 2.34 milestone Aug 3, 2017

kalikiana changed the title from lxd: Snapcraft update in containers to lxd: snapcraft update in containers Aug 8, 2017

kalikiana added some commits Aug 8, 2017

sergiusens referenced this pull request Aug 9, 2017

update story #1462

sergiusens changed the title from lxd: snapcraft update in containers to lxd: snapcraft refresh in containers Aug 30, 2017

sergiusens reviewed Sep 1, 2017

View changes

sergiusens modified the milestones: 2.35, 2.34 Sep 6, 2017

kalikiana added some commits Sep 7, 2017

kyrofa reviewed Sep 8, 2017

View changes

kalikiana added some commits Sep 12, 2017

kyrofa commented Sep 12, 2017 •

Edited 1 time

kyrofa
Sep 12, 2017

kyrofa dismissed their stale review Sep 12, 2017

kyrofa commented Sep 12, 2017

kalikiana added some commits Sep 13, 2017

kalikiana dismissed sergiusens’s stale review Oct 4, 2017

kalikiana referenced this pull request Oct 12, 2017

cli: update parts cache in the container #1546

kalikiana commented Oct 20, 2017

kalikiana commented Oct 23, 2017

kalikiana closed this Oct 23, 2017

kyrofa commented Oct 27, 2017

kalikiana reopened this Nov 3, 2017

sergiusens commented Nov 7, 2017

kalikiana commented Nov 7, 2017 •

Edited 1 time

kalikiana
Nov 7, 2017

sergiusens reviewed Nov 8, 2017

View changes

kalikiana and others added some commits Nov 8, 2017

sergiusens approved these changes Nov 8, 2017

View changes

sergiusens merged commit `3d511c5` into snapcore:master Nov 9, 2017
1 check passed

1 check passed

matiasb added a commit to matiasb/snapcraft that referenced this pull request Nov 24, 2017

snapcore/snapcraft

Join GitHub today

lxd: snapcraft refresh in containers #1412

Conversation

kalikiana commented Jul 17, 2017 • edited Edited 1 time kalikiana Aug 25, 2017

kalikiana referenced this pull request Jul 17, 2017

kalikiana self-assigned this Jul 19, 2017

kalikiana requested a review from sergiusens Jul 19, 2017

kalikiana added some commits Jul 27, 2017

sergiusens added this to the 2.34 milestone Aug 3, 2017

kalikiana changed the title from lxd: Snapcraft update in containers to lxd: snapcraft update in containers Aug 8, 2017

kalikiana added some commits Aug 8, 2017

sergiusens referenced this pull request Aug 9, 2017

sergiusens changed the title from lxd: snapcraft update in containers to lxd: snapcraft refresh in containers Aug 30, 2017

sergiusens reviewed Sep 1, 2017 View changes

sergiusens modified the milestones: 2.35, 2.34 Sep 6, 2017

kalikiana added some commits Sep 7, 2017

kyrofa reviewed Sep 8, 2017 View changes

kalikiana added some commits Sep 12, 2017

kyrofa commented Sep 12, 2017 • edited Edited 1 time kyrofa Sep 12, 2017

kyrofa dismissed their stale review Sep 12, 2017

kyrofa commented Sep 12, 2017

kalikiana added some commits Sep 13, 2017

kalikiana dismissed sergiusens’s stale review Oct 4, 2017

kalikiana referenced this pull request Oct 12, 2017

kalikiana commented Oct 20, 2017

kalikiana commented Oct 23, 2017

kalikiana closed this Oct 23, 2017

kyrofa commented Oct 27, 2017

kalikiana reopened this Nov 3, 2017

sergiusens commented Nov 7, 2017

kalikiana commented Nov 7, 2017 • edited Edited 1 time kalikiana Nov 7, 2017

sergiusens reviewed Nov 8, 2017 View changes

kalikiana and others added some commits Nov 8, 2017

sergiusens approved these changes Nov 8, 2017 View changes

Hide details View details sergiusens merged commit 3d511c5 into snapcore:master Nov 9, 2017 1 check passed

1 check passed

matiasb added a commit to matiasb/snapcraft that referenced this pull request Nov 24, 2017

kalikiana commented Jul 17, 2017 •

Edited 1 time

kalikiana
Aug 25, 2017

sergiusens reviewed Sep 1, 2017

View changes

kyrofa reviewed Sep 8, 2017

View changes

kyrofa commented Sep 12, 2017 •

Edited 1 time

kyrofa
Sep 12, 2017

kalikiana commented Nov 7, 2017 •

Edited 1 time

kalikiana
Nov 7, 2017

sergiusens reviewed Nov 8, 2017

View changes

sergiusens approved these changes Nov 8, 2017

View changes

sergiusens merged commit `3d511c5` into snapcore:master Nov 9, 2017
1 check passed