/var/lib/lxd/containers/<name>/rootfs only exists when container is running.

[smoser]

Required information

• Distribution: Ubuntu

• Distribution version: 17.10

• The output of "lxc info" or if that fails:

  config: {}
  api_extensions:
  - storage_zfs_remove_snapshots
  - container_host_shutdown_timeout
  - container_syscall_filtering
  - auth_pki
  - container_last_used_at
  - etag
  - patch
  - usb_devices
  - https_allowed_credentials
  - image_compression_algorithm
  - directory_manipulation
  - container_cpu_time
  - storage_zfs_use_refquota
  - storage_lvm_mount_options
  - network
  - profile_usedby
  - container_push
  - container_exec_recording
  - certificate_update
  - container_exec_signal_handling
  - gpu_devices
  - container_image_properties
  - migration_progress
  - id_map
  - network_firewall_filtering
  - network_routes
  - storage
  - file_delete
  - file_append
  - network_dhcp_expiry
  - storage_lvm_vg_rename
  - storage_lvm_thinpool_rename
  - network_vlan
  - image_create_aliases
  - container_stateless_copy
  - container_only_migration
  - storage_zfs_clone_copy
  - unix_device_rename
  - storage_lvm_use_thinpool
  - storage_rsync_bwlimit
  - network_vxlan_interface
  - storage_btrfs_mount_options
  - entity_description
  - image_force_refresh
  - storage_lvm_lv_resizing
  - id_map_base
  - file_symlinks
  - container_push_target
  - network_vlan_physical
  - storage_images_delete
  - container_edit_metadata
  - container_snapshot_stateful_migration
  - storage_driver_ceph
  - storage_ceph_user_name
  - resource_limits
  - storage_volatile_initial_source
  api_status: stable
  api_version: "1.0"
  auth: trusted
  public: false
  environment:
    addresses: []
    architectures:
    - x86_64
    - i686
    certificate: |
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----
    certificate_fingerprint: 79835e469ddd6eb1c253801988b8b950bbe87e83c1d21ee8bea8ff2b6ec4df95
    driver: lxc
    driver_version: 2.0.8
    kernel: Linux
    kernel_architecture: x86_64
    kernel_version: 4.12.0-11-generic
    server: lxd
    server_pid: 14589
    server_version: "2.17"
    storage: zfs
    storage_version: 0.6.5.11-1ubuntu3
  

Issue description

the directory /var/lib/lxd/containers//rootfs only exists when the container is running.
This makes some of our work flows based on 'lxd init ... change contents ... lxd start' not work correctly, specifically with 'mount-image-callback'

1. lxd launch ubuntu-daily:xenial x1

2. look for directory in expected path.

   $ sudo ls /var/lib/lxd/containers/x2/rootfs
   ls: cannot access '/var/lib/lxd/containers/x2/rootfs': No such file or directory
   

Alternatively, try 'mount-image-callback lxd:x2 mchroot`

Information to attach

• any relevant kernel output (dmesg)
• container log (lxc info NAME --show-log)
• main daemon log (cat /var/log/lxd/lxd.log)
• output of the client with --debug
• output of the daemon with --debug

[stgraber]

Yes, that is expected behavior. Keeping all the containers mounted was causing a number of kernel issues, so we're only mounting things on demand now.

Direct access to the container's filesystem at /var/lib/lxd/containers/NAME/rootfs has never been part of any documented/supported way of interacting with LXD.

Interactions with the container's filesystem should be done through our file management API or through an exec session after the container is started, anything else isn't supported and is subject to changes.

In your case, from what I remember of what you're doing, the easiest option is probably for you to just manually mount the container before doing any changes. You can do so with "zfs mount" and "zfs unmount". The relevant paths can be found in "zfs list -t all".

[brauner]

Is zfs mount -a and then zfs umount -a an option? That would mount all filesystems though.

[stgraber]

I don't believe "zfs mount -a" mounts the filesystems that are flagged as "canmount=false".
Either that or something other than "zfs mount -a" is used at boot time.

In any case, I wouldn't recommend mounting everything because that's just asking for the zfs EBUSY kernel bug to bite you. You should really only ever mount what you need and nothing more, making sure that things are unmounted before starting the container so that we avoid duplicate mount table entries that confuse the heck out of zfs.

[smoser]

the other issue with zfs (or any other backend) is that i have to have backend specific knowledge and determine which backend is in use.

[stgraber]

Yep, but that part isn't new. We've never kept containers mounted when they're on LVM or Ceph.

[smoser]

I guess i'd argue that any problems you had due to "Keeping all the containers mounted" are still inevitable. If there are kernel issues then you just pushed finding these issues to when a user starts running a bunch of containers all at once rather than just having a bunch of containers at rest (and who does that?)

[stgraber]

Not really. The problem we had was that if containers were auto-mounted, then daemons starting at boot time (snapd for example) would potentially fork your mount table, confusing ZFS among other things.

With them only being mounted immediately before the container's mount namespace is created, this avoids that problem entirely.

Believe me, we don't like doing those kind of annoying multi-release changes just because it's fun. We do them because we've confirmed that they do fix a problem.

[jfgibbins]

Was that a recent change? Cause I used to ls through those directories all the time. At what density did you start to see issues? I've been up around 40-50, but hadn't noticed anything. BTW, Stephane, Christian and Scott all in one thread, scary!

[stgraber]

We've seen issues with as low as 4-5 containers, the problem tended to be more closely related to the number of mount namespaces, when those were created and whether the kernel would reap them quickly enough, more than the number of containers (number of containers would make the mount table bigger, making it take longer to flush).

[stgraber]

And those directories will still work fine for most users, so long as the container is running. It's only when it's not running that we keep it unmounted.