[Arch|AUR] After upgrading to 3.4 nginx won't start on container because it can't open /dev/null

[C0rn3j]

Required information

• Distribution: Arch
• The output of "lxc info":

api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIIFUjCCAzqgAwIBAgIQXpQmDnPV7dRRPauSk0aC/zANBgkqhkiG9w0BAQsFADA0
    MRwwGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QEFj
    ZWRpYTAeFw0xODAyMjExNjQ0MjFaFw0yODAyMTkxNjQ0MjFaMDQxHDAaBgNVBAoT
    E2xpbnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAQWNlZGlhMIICIjAN
    BgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAye4mGK2TfYtMCYn9/SJClGjN3o75
    vsRApKtyNwuwPY8A6Rktyef5o/n2mZIrlZ3a92MCGt6keC6SlbwIkTw7Xefe4id5
    NNsGwbVQyKOUOIEPnkbUjxZS520FcOGdbY4IutrFY0H3XhGYZ0lwQnHwkAp2h2kf
    3N7Z8Oxz/S3zHMtIzZ3w0citnuyF/vyrlc8oiz7yU0DWoF7N6JVERKyP2UsNxUtT
    LMStq9Mk5w8m/DUgQqXGvq1kqeNzNiwAbMLYNo5TGtH/wXQfXkVQPTmuas3WnxQr
    WgeRnc132uKCHLqte96d9ojMNMOFd+gsyiI9uNSXloV+mqaQiWYe4/L/BrxlgXln
    1H+o4Q6sz/NDtsq3Itvk9gubVTxx+/TIxis+6RegqQC9II6HCXLLNoqTvuC/xdKq
    cHsv7NHVDYtz4qnKBuTVIebrjZfoPva+mr4ZTJVboOA/VmnZ7p+DUM7pAqHGKaF2
    aaNttb+XiRn7tA6H/tQ1Cru5tw6Gm4mKQx/A5twKIkkgEqKZINpTmxs2qTCB+Gmo
    mpz2qKFaA2wvZv/BUnJaUsoGRMrITQ587imqnyYA1pBNbDwvj2PM7cWfee1ojOdK
    wsHSfpjMNLJA2QDZPjPfXtiae82sYQRsxeGkHtLsjkSOvBJ/QEgbShfINUDcbESB
    UZFzsGw19XiHgp0CAwEAAaNgMF4wDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoG
    CCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwKQYDVR0RBCIwIIIGQWNlZGlhhwTAqAEK
    hxAqABAolsr+dkh1Rv/+hLMtMA0GCSqGSIb3DQEBCwUAA4ICAQCKKxWkxOXoT3ul
    kmZ3Wx/H59GsUJpZQu0LCkBvEMeKW4fSz97hQPZ1GLOGGnL4bOcEtYM5mBJgzND6
    Df9WO6IjVZ28yWVqZGr74kAoXIBnXIyMn9UtvXH7B2jY5Zjc8Ku3hP073oTYF1E/
    DHQV4l3tmR31ys+/CXwEQc+YShljxLimA6/dqnPJKm6gcAO/Q0pg8AI4Q93ps7H9
    eGw6je8Yn81HrXOn+hOprYNo+SdWsVsk/GD1Ath3R+oxeQbgwDTokht0K97PUxyz
    tJSeii+zjgCO9lJ5ytC/s7CRARMK7mxWAXm8OL7qW+AhIMy41q7zvvRy1FQqCQYh
    k4mU8e2nvbsK09jzqb+GtrhG2K74ejgGFuCv3OuOFL7Pen9G59346jmPLitW+Edd
    0oFUkleqO7RBKGN+V1uN6FE9u2lHaiQ17nM1WZbYBaesJPjoiWwbyKb2dcAv3g7v
    FaX0vRUy/BgqngGIMxESRLBqDeWuYwLGoSYlb8Ix+DDnpUYrhuadoy//ITXAJip0
    i0Ghl4yfF20EZ5ib3OENEbmCbBCSichhlBZcYYyh7WuY3FHNjNWW638RlSpe7jbs
    Ymzu9YxG1+IzlOhKeNzVdduOb9DpEE5cm5ekiWSUhsSCFbHWkEtubAHcRC4CjQAZ
    URDBjanfqSgrpQu3LGr8wUvUu9+OoA==
    -----END CERTIFICATE-----
  certificate_fingerprint: 214a2bdb32e49a05b33c75dc65b02d5c3bf051098770543c5ad447009263d730
  driver: lxc
  driver_version: 3.0.2
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.18.1-arch1-1-ARCH
  server: lxd
  server_pid: 338
  server_version: "3.4"
  storage: btrfs
  storage_version: 4.17.1
  server_clustered: false
  server_name: Acedia

Issue description

After update from 3.3 to 3.4 (I use the AUR lxd package) containers won't run nginx.

Aug 19 21:19:09 proxy systemd[1]: nginx.service: Failed to reset devices.list: Operation not permitted
Aug 19 21:19:09 proxy systemd[1]: Starting A high performance web server and a reverse proxy server...
Aug 19 21:19:09 proxy nginx[2063]: 2018/08/19 21:19:09 [warn] 2063#2063: could not build optimal types_hash, you should increase either types_hash_max_size: 1024 or types_hash_bucket_size: 64; ignoring types_hash_bucket_size
Aug 19 21:19:09 proxy nginx[2063]: 2018/08/19 21:19:09 [emerg] 2064#2064: open("/dev/null") failed (13: Permission denied)
Aug 19 21:19:09 proxy systemd[1]: nginx.service: Can't open PID file /run/nginx.pid (yet?) after start: No such file or directory

Note: At the same time as I upgraded I managed to get IPv6 finally working on my network, so it could possibly be related.

Steps to reproduce

1. Upgrade to 3.4
2. Create an Arch container, install nginx-mainline and try to start nginx on it
3. It fails with a /dev/null error

Workaround

Setting the container to a privileged one works around this.

[stgraber]

It's because you also switched to the 4.18 kernel which changes the way mknod is handled.

Either downgrade to an earlier kernel or upgrade liblxc to 3.0.2 which contains a workaround to deal with that kernel behavior change.

[C0rn3j]

From my info:

  driver: lxc
  driver_version: 3.0.2

I'd imagine this is the liblxc version? If not then am not sure how I'd go about upgrading it as I can't find any reference to it in the PKGBUILD https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=lxd

How do I even get which liblxc version I am using if it's not in the driver_version row I posted?

[brauner]

@stgraber, this is LXD running with liblxc 3.0.2 as the output shows. LXC is handling this case correctly otherwise the container wouldn't run.

This is caused by systemd's PrivateDevices feature which isn't yet able to deal with vfs_mknod() in user namespaces. Fwiw, this is caused by https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=55956b59df336f6738da916dbb520b6e37df9fbd . I discussed reverting this with @ebiederm but it was decided to move forward with this change.

I tried to fix this regression in the kernel as well as in systemd. Since we've decided to move forward with this change on the kernel side you should report this failure on my systemd PR ( systemd/systemd#9483 ) and lobby for a systemd change if they are open for it.