userns-remap enabled - wrong uids/gids in tar archive generated by command docker cp -a container:/path/to/dir -

[nthproprio]

Description
My docker daemon has the userns-remap feature enabled.
When I run command docker cp -a my-container:/path/to/dir - > /tmp/copy.tar, files and directories within generated archive have wrong UID/GID (for example, files owned by user 1000 in container will have owner 166536). It seems UIDs/GIDs considered when creating tar archive are the host ids and not the container ones we can see from inside container.
However, when I run docker cp -a - my-container:/tmp/ < /tmp/copy.tar to copy directory from archive to container, it fails with an error like below :

Error response from daemon: Error processing tar file(exit status 1): Container ID 166536 cannot be mapped to a host ID

It seems to consider UIDs/GIDs from given archive as Container IDs and not host IDs, and so it fails because there are only 65536 sub-uids/sub-gids allocated for user dockremap.

Steps to reproduce the issue:

1 Configuration on host with userns-remap feature enabled

user@ubuntu:~$ cat /etc/docker/daemon.json 
{
  "debug": true,
  "userns-remap": "default"
}

user@ubuntu:~$ cat /etc/subuid
user:100000:65536
dockremap:165536:65536

user@ubuntu:~$ cat /etc/subgid
user:100000:65536
dockremap:165536:65536

2. Run container

user@ubuntu:~$ docker run --name elasticsearch --detach --volume /usr/share/elasticsearch/data -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" elasticsearch:7.17.4
Unable to find image 'elasticsearch:7.17.4' locally
7.17.4: Pulling from library/elasticsearch
d5fd17ec1767: Pull complete 
3aceae0816c1: Pull complete 
6f282e391d7d: Pull complete 
e0d1c86ab271: Pull complete 
1c2d02571b2b: Pull complete 
25fb4b01f643: Pull complete 
606786004049: Pull complete 
28ec7712324b: Pull complete 
7d5976c54116: Pull complete 
Digest: sha256:529b3cfec4354beda158c6c7f2f8015cbdc9432a48c1d63e824d6fd728f30db2
Status: Downloaded newer image for elasticsearch:7.17.4
c11f288143e8375bee899588bd5189d1c479a460aa1762a5d1d1bd39cfdbd219

3. directory /usr/share/elasticsearch/data/nodes has uid 1000 as owner

user@ubuntu:~$ docker exec elasticsearch stat /usr/share/elasticsearch/data/nodes
  File: /usr/share/elasticsearch/data/nodes
  Size: 4096      	Blocks: 8          IO Block: 4096   directory
Device: fd00h/64768d	Inode: 1186361     Links: 3
Access: (0775/drwxrwxr-x)  Uid: ( 1000/elasticsearch)   Gid: (    0/    root)
Access: 2022-06-05 08:17:59.409081230 +0000
Modify: 2022-06-05 08:17:59.409081230 +0000
Change: 2022-06-05 08:17:59.409081230 +0000
 Birth: -

4. Stop container

user@ubuntu:~$ docker stop elasticsearch 
elasticsearch

5. Copy directory /usr/share/elasticsearch/data from container to an archive file

user@ubuntu:~$ docker cp -a elasticsearch:/usr/share/elasticsearch/data/ - > ./backup.tar

6. Extract directory data from archive to /tmp in container

user@ubuntu:~$ docker cp -a - elasticsearch:/tmp/ < ./backup.tar
Error response from daemon: Error processing tar file(exit status 1): Container ID 166536 cannot be mapped to a host ID

Stat on directory nodes from archive content

user@ubuntu:~$ sudo tar -xf backup.tar -C ./
user@ubuntu:~$ stat data/nodes/
  File: data/nodes/
  Size: 4096      	Blocks: 8          IO Block: 4096   directory
Device: fd00h/64768d	Inode: 1186335     Links: 3
Access: (0775/drwxrwxr-x)  Uid: (166536/ UNKNOWN)   Gid: (165536/ UNKNOWN)
Access: 2022-06-05 08:22:08.262565926 +0000
Modify: 2022-06-05 08:17:59.000000000 +0000
Change: 2022-06-05 08:22:08.310565453 +0000
 Birth: -

Describe the results you received:
In generated archive, UIDs/GIDs are Host IDs

Describe the results you expected:
In generated archive, UIDs/GIDs set for files/directories should be Container IDs. In example above, directory nodes should have uid 1000 as owner and not 166536.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.2
 Git commit:        20.10.12-0ubuntu2~20.04.1
 Built:             Wed Apr  6 02:14:38 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.2
  Git commit:       20.10.12-0ubuntu2~20.04.1
  Built:            Thu Feb 10 15:03:35 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.9-0ubuntu1~20.04.4
  GitCommit:        
 runc:
  Version:          1.1.0-0ubuntu1~20.04.1
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        

Output of docker info:

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 
 runc version: 
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: default
  userns
 Kernel Version: 5.4.0-113-generic
 Operating System: Ubuntu 20.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 5.804GiB
 Name: ubuntu
 ID: 5FNU:JBDW:K3VU:JBFZ:LMW4:PGZP:BJSA:ISMF:ZDFD:SRJO:NVMJ:5GVR
 Docker Root Dir: /var/lib/docker/165536.165536
 Debug Mode: true
  File Descriptors: 24
  Goroutines: 34
  System Time: 2022-06-05T08:16:33.521794354Z
  EventsListeners: 0
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):