docker commit > invalid tar header #33586

SeidChr · 2017-06-08T10:36:19Z

Description

Hi,

After importing a couple of databases in the ms developer sql database image, then stopping it, and committing, i get the following error:

PS > docker commit ContainerXY my/tagError 
response from daemon: re-exec error: exit status 1: output: archive/tar: invalid tar header

The Databases are quite big though, but anyway, it should not fail like that.
Is there any more info i can give you to solve that issue?

Steps to reproduce the issue:

Start ms developer sql database image
import 120gig of databases (final findows filter directory size)
commit the image

Describe the results you received:

response from daemon: re-exec error: exit status 1: output: archive/tar: invalid tar header

Describe the results you expected:
image comitted correctly

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version & docker info:

PS C:\Windows\system32> docker info
Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 209
Server Version: 17.03.1-ee-3
Storage Driver: windowsfilter
 Windows: 
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: l2bridge l2tunnel nat null overlay transparent
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.1198.amd64fre.rs1_release_sec.170427-1353)
Operating System: Windows Server 2016 Standard
OSType: windows
Architecture: x86_64
CPUs: 12
Total Memory: 63.92 GiB
Name: S6
ID: 2OZY:DSZI:JEUU:4ISW:GJ23:F2TA:F5U3:XR4Z:YZKU:SAID:362W:MMXD
Docker Root Dir: D:\System\Data\Docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: -1
 Goroutines: 22
 System Time: 2017-06-02T09:56:38.4659153+02:00
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

PS C:\Windows\system32> docker version
Client:
 Version:      17.03.1-ee-3
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   3fcee33
 Built:        Thu Mar 30 19:31:22 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.03.1-ee-3
 API version:  1.27 (minimum version 1.24)
 Go version:   go1.7.5
 Git commit:   3fcee33
 Built:        Thu Mar 30 19:31:22 2017
 OS/Arch:      windows/amd64
 Experimental: false

PS C:\Windows\system32>

Additional environment details (AWS, VirtualBox, physical, etc.):
Windows Server 2016 and updated docker
https://social.msdn.microsoft.com/Forums/en-US/14d2ad0c-1116-42c7-8449-29d495be4f30/docker-commit-invalid-tar-header?forum=windowscontainers
Screenshot of windowsfilter folder properties https://social.msdn.microsoft.com/Forums/getfile/1080800

The text was updated successfully, but these errors were encountered:

thaJeztah · 2017-06-08T10:38:55Z

/cc @jstarks @johnstep

snorrk · 2017-10-13T01:18:03Z

I'm hitting the same issue, it seems to be related to the size of the container.

Steps

$containerName = "Test01"

docker run -d --name $containerName  --volume c:\shared:c:\shared --storage-opt size=60G --memory 10g microsoft/iis

$containerInfo = docker container inspect $containerName | ConvertFrom-Json
Enter-PSSession -ContainerID $containerInfo.id -RunAsAdministrator

copy c:\shared\18-gigabyte-file.dat c:\

exit

docker stop container $containerName
docker commit $containerName testcontainerimage

Result

Error response from daemon: re-exec error: exit status 1: output: archive/tar: invalid tar header

Environment

docker version

Client:
 Version:      17.06.2-ee-3
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   915cbaa
 Built:        Thu Sep 21 23:01:20 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.06.2-ee-3
 API version:  1.30 (minimum version 1.24)
 Go version:   go1.8.3
 Git commit:   915cbaa
 Built:        Thu Sep 21 23:10:04 2017
 OS/Arch:      windows/amd64
 Experimental: false

docker info

Containers: 4
 Running: 2
 Paused: 0
 Stopped: 2
Images: 3
Server Version: 17.06.2-ee-3
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd json-file logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.1715.amd64fre.rs1_release_inmarket.170906-1810)
Operating System: Windows Server 2016 Datacenter
OSType: windows
Architecture: x86_64
CPUs: 4
Total Memory: 16GiB
Name: snbeck-test01
ID: Z2FF:YVBZ:BI7Y:FD2Y:2LY4:46VW:FVYT:UAD5:57WS:UQOF:2IT4:YAQC
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

thaJeztah · 2017-10-14T21:06:51Z

@n4ss @stevvooe related to #35075, or different code path?

JanneRantala · 2017-10-17T08:02:57Z

Same issue here with Windows 10 Creators Update and latest Docker from edge channel. I was trying to restore 1.5 gig database backup file to SQL Server when got that error. Complete db size would have been around 15 gigs.

Client:
Version: 17.09.0-ce
API version: 1.32
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:40:09 2017
OS/Arch: windows/amd64

Server:
Version: 17.09.0-ce
API version: 1.32 (minimum version 1.24)
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:50:27 2017
OS/Arch: windows/amd64
Experimental: true

thaJeztah · 2017-10-17T11:05:44Z

ping @johnstep PTAL

snorrk · 2017-10-19T03:02:06Z

For what it's worth, I also had SQL Server and database files in my container.

Looking at #35075, I believe that SQL database files are expanded to be of a specific size (presumably with the extra space zeroed out) so it is very possible that this is issue is similar to that one.

dmarkle · 2017-12-16T13:24:18Z

I can repro this on a SQL 2017 (microsoft/mssql-server-windows-developer) restoring a ~30GB database. I'm seeing this particular error manifest itself when I docker-compose build my test environment. The RUN for the database restore works fine and I get the standard SQL Server message saying that I've processed 3763487 pages,, but once that's done, I get
ERROR: Service 'test_service' failed to build: re-exec error: exit status 1: output : archive/tar: invalid tar header

Version: 17.09.1-ce
API version 1.32
OS/arch windows/amd64

thaJeztah · 2017-12-16T20:12:47Z

Does it work if you set the size storage option for the daemon to a bigger value? (I think the default may be 20 or 30G); https://github.com/docker/cli/blob/master/docs/reference/commandline/dockerd.md

Easiest way to do so is to create a daemon.json configurationfile for this (see the linked document); typing from my phone so can't easily give a full example, but happy to help.

After changing the configuration, the dockerd daemon must be restarted

dmarkle · 2017-12-16T20:17:05Z

No. I had to set the size storage option and restart dockerd to even get to this point. If I hadn’t made the storage-opt adjustment, I would have just received an error restoring the database complaining that there was not enough space on disk.

…

On Sat, Dec 16, 2017 at 3:13 PM Sebastiaan van Stijn < ***@***.***> wrote: Does it work if you set the size storage option for the daemon to a bigger value? (I think the default may be 20 or 30G); https://github.com/docker/cli/blob/master/docs/reference/commandline/dockerd.md Easiest way to do so is to create a daemon.json configurationfile for this (see the linked document); typing from my phone so can't easily give a full example, but happy to help. After changing the configuration, the dockerd daemon must be restarted — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#33586 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADrezZnY6kUa8k5uPRS3dj7K_GejZY-Xks5tBCSDgaJpZM4Nz4ig> .

thaJeztah · 2017-12-16T20:22:14Z

Thanks for the extra information; I was wondering if the "invalid tar header" would have been due to that (and just an incorrect / confusing error)

mikebolland · 2018-01-03T10:29:25Z

We are also experiencing this issue dockerising oracle and windows server core as the base image

 ---> Running in a2e00d5102f8
Copying database files
1% complete
3% complete
11% complete
18% complete
26% complete
33% complete
37% complete
Creating and starting Oracle instance
40% complete
45% complete
50% complete
55% complete
56% complete
60% complete
62% complete
Completing Database Creation
66% complete
70% complete
73% complete
85% complete
96% complete
100% complete
re-exec error: exit status 1: output: archive/tar: invalid tar header

We have changed the daemon settings to 100G to allow the dbca command above to complete but the image push fails

{
  "registry-mirrors": [],
  "insecure-registries": [],
  "debug": true,
  "storage-opts": [
    "size=100G"
  ],
  "experimental": true
}

Z:\>docker version
Client:
 Version:      17.09.1-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:22:26 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.09.1-ce
 API version:  1.32 (minimum version 1.24)
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:32:40 2017
 OS/Arch:      windows/amd64
 Experimental: true

Z:\>docker version
Client:
 Version:      17.09.1-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:22:26 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.09.1-ce
 API version:  1.32 (minimum version 1.24)
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:32:40 2017
 OS/Arch:      windows/amd64
 Experimental: true

Z:\>docker info
Containers: 3
 Running: 1
 Paused: 0
 Stopped: 2
Images: 7
Server Version: 17.09.1-ce
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: ics l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd json-file logentries splunk syslog
Swarm: inactive
Default Isolation: hyperv
Kernel Version: 10.0 14393 (14393.1358.amd64fre.rs1_release.170602-2252)
Operating System: Windows 10 Pro
OSType: windows
Architecture: x86_64
CPUs: 8
Total Memory: 15.95GiB
Name: PCSL1004
ID: 7H3J:EYJL:GWPB:GJST:XDXY:S5FT:KA73:2XIS:TVXP:2PR5:YDYR:M7VW
Docker Root Dir: D:\ProgramData\Docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: -1
 Goroutines: 41
 System Time: 2018-01-03T10:48:15.0758851Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

thaJeztah · 2018-01-03T11:35:02Z

@johnstep I think you mentioned that this could possibly be due to https://github.com/Microsoft/go-winio using an older version of archive/tar (based on Go 1.6)? https://github.com/Microsoft/go-winio/tree/424d62c72ec0db3fa2b19791b6f96c7eb5a5e412/archive/tar

johnstep · 2018-01-03T23:11:43Z

Yes, apparently the 8 GB restriction was removed in Go 1.8. We need to review whether that modified copy of archive/tar in go-winio is still required and, if so, update it; otherwise, remove it and switch to the standard runtime package. I spoke with @jhowardmsft separately, as well.

mikebolland · 2018-01-04T13:34:42Z

I notice most of the posts above have remapped docker to d:\ (in my case symlinked c:\programdata to d:\programdata) so i have tested this on docker-ee 17.9 with a very large c:\ partition to confirm windows mklink is not the cause

Sadly the result is the same
re-exec error: exit status 1: output: archive/tar: invalid tar header

mikebolland · 2018-01-04T18:43:05Z

Is the attempt to fix this being done as part of #35739 ?

ElasticCoder · 2018-02-09T14:14:14Z

I have the same problem. Restoring a 15GB database into SQLExpress.

RESTORE DATABASE successfully processed 839885 pages in 32.105 seconds (204.379 MB/sec).
re-exec error: exit status 1: output: archive/tar: invalid tar header

PS C:\Source Code\Db-container> docker image ls
REPOSITORY                               TAG                 IMAGE ID            CREATED             SIZE
<none>                                   <none>              ebaf1a726468        3 minutes ago       15.2GB

TheHackmeister · 2018-04-23T17:51:06Z

Just wanted to drop a line saying I'm experiencing this issue in 18.03 stable. I'm trying to import a DB that ends up taking 25ish GB. It run fine (once I increased storage-opts), but as soon as I try to save the container to an image, I get the invalid tar header error. This is the case when I build build the container, as well as when I manually run the steps in a container and then trying to commit said container.

It sounds like we have a possible solution, but if you need any more information or help, I'm game to experiment and would love to get this working. Just let me know how I can help.

Windows 10: 1607 (Build 14393.2125)

C:\inetpub\wwwroot>docker version
Client:
 Version:       18.03.0-ce
 API version:   1.37
 Go version:    go1.9.4
 Git commit:    0520e24
 Built: Wed Mar 21 23:06:28 2018
 OS/Arch:       windows/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:      18.03.0-ce
  API version:  1.37 (minimum version 1.24)
  Go version:   go1.9.4
  Git commit:   0520e24
  Built:        Wed Mar 21 23:21:06 2018
  OS/Arch:      windows/amd64
  Experimental: false

C:\inetpub\wwwroot>docker info
Containers: 4
 Running: 2
 Paused: 0
 Stopped: 2
Images: 42
Server Version: 18.03.0-ce
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: ics l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd gelf json-file logentries splunk syslog
Swarm: inactive
Default Isolation: hyperv
Kernel Version: 10.0 14393 (14393.2125.amd64fre.rs1_release.180301-2139)
Operating System: Windows 10 Enterprise
OSType: windows
Architecture: x86_64
CPUs: 4
Total Memory: 15.66GiB
Name: shachmeist1217
ID: 3GO5:EVIM:P2QZ:7PGM:VNSV:7GTK:MOWX:NHP2:43GP:6DNM:57IN:MXGR
Docker Root Dir: C:\ProgramData\Docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: -1
 Goroutines: 41
 System Time: 2018-04-23T11:47:18.7785299-06:00
 EventsListeners: 1
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Registry Mirrors:
 https://peakdockerregistry.azurecr.io/
Live Restore Enabled: false

viniciusxyz · 2018-08-31T12:19:26Z

Alguma atualização para o problema? Estou com o mesmo erro, segue as informações do docker:

Contêineres: 8 Em
execução: 0
Em pausa: 0
Parado: 8
Imagens: 1
Versão do servidor: 17.06.2-ee-15
Driver de armazenamento: windowsfilter
Windows:
Logging Driver: json-file
Plugins:
Volume: local
Rede: l2bridge l2tunnel nat overlay nulo transparente
Log: awslogs etwlogs fluentd json-arquivo logentries splunk syslog
Swarm: inativo
Default Isolation: process
Versão do Kernel: 10.0 14393 (14393.2248.amd64fre.rs1_release.180427-1804)
Sistema operacional: Windows Server 2016
OSType: windows
Arquitetura: x86_64
CPUs: 8
Memória Total: 31.89GiB
Nome: AUTKSVRH10
ID: QJEV: UXTP: RJVJ: 4MS2: M777: BSNM: CILP: Z62Y: 5UQJ: UMSY: C5CP: Roteador de ancoragem BLK7
Dir: C: \ ProgramData \ docker
Modo de depuração (cliente): false
Modo de depuração (servidor): false
Registro : https://index.docker.io/v1/
Experimental: false
Registros Inseguros:
140.1.254.10:5000
127.0.0.0/8
Live Restore Ativado: false

smasherprog · 2019-04-13T03:00:51Z

I have this problem as well, is this going to be fixed at all?

Daniel-Glucksman · 2020-03-18T07:59:19Z

Same here

TBBle · 2020-11-05T07:41:15Z

I suspect this was fixed in #41430 (for Docker 20.10.0), assuming #33586 (comment) was an accurate diagnosis.

thaJeztah · 2020-11-05T10:59:48Z

Ah, yes, I guess this was fixed by #41430 / microsoft/go-winio#175

Let me go ahead and close

GordonTheTurtle added platform/windows version/17.03 labels Jun 8, 2017

mikebolland mentioned this issue Jan 3, 2018

Archive/tar: invalid tar header message when export or import docker/for-win#660

Closed

thaJeztah assigned johnstep Jan 3, 2018

mikebolland mentioned this issue Jan 5, 2018

Windows: Bump RW layer size #35925

Merged

thaJeztah closed this as completed Nov 5, 2020

thaJeztah added the kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. label Nov 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker commit > invalid tar header #33586

docker commit > invalid tar header #33586

SeidChr commented Jun 8, 2017 •

edited

thaJeztah commented Jun 8, 2017

snorrk commented Oct 13, 2017 •

edited by thaJeztah

thaJeztah commented Oct 14, 2017

JanneRantala commented Oct 17, 2017 •

edited

thaJeztah commented Oct 17, 2017

snorrk commented Oct 19, 2017 •

edited

dmarkle commented Dec 16, 2017 •

edited

thaJeztah commented Dec 16, 2017

dmarkle commented Dec 16, 2017 via email

thaJeztah commented Dec 16, 2017

mikebolland commented Jan 3, 2018 •

edited

thaJeztah commented Jan 3, 2018

johnstep commented Jan 3, 2018

mikebolland commented Jan 4, 2018

mikebolland commented Jan 4, 2018

ElasticCoder commented Feb 9, 2018 •

edited

TheHackmeister commented Apr 23, 2018

viniciusxyz commented Aug 31, 2018

smasherprog commented Apr 13, 2019

Daniel-Glucksman commented Mar 18, 2020

TBBle commented Nov 5, 2020 •

edited

thaJeztah commented Nov 5, 2020

docker commit > invalid tar header #33586

docker commit > invalid tar header #33586

Comments

SeidChr commented Jun 8, 2017 • edited

thaJeztah commented Jun 8, 2017

snorrk commented Oct 13, 2017 • edited by thaJeztah

thaJeztah commented Oct 14, 2017

JanneRantala commented Oct 17, 2017 • edited

thaJeztah commented Oct 17, 2017

snorrk commented Oct 19, 2017 • edited

dmarkle commented Dec 16, 2017 • edited

thaJeztah commented Dec 16, 2017

dmarkle commented Dec 16, 2017 via email

thaJeztah commented Dec 16, 2017

mikebolland commented Jan 3, 2018 • edited

thaJeztah commented Jan 3, 2018

johnstep commented Jan 3, 2018

mikebolland commented Jan 4, 2018

mikebolland commented Jan 4, 2018

ElasticCoder commented Feb 9, 2018 • edited

TheHackmeister commented Apr 23, 2018

viniciusxyz commented Aug 31, 2018

smasherprog commented Apr 13, 2019

Daniel-Glucksman commented Mar 18, 2020

TBBle commented Nov 5, 2020 • edited

thaJeztah commented Nov 5, 2020

SeidChr commented Jun 8, 2017 •

edited

snorrk commented Oct 13, 2017 •

edited by thaJeztah

JanneRantala commented Oct 17, 2017 •

edited

snorrk commented Oct 19, 2017 •

edited

dmarkle commented Dec 16, 2017 •

edited

mikebolland commented Jan 3, 2018 •

edited

ElasticCoder commented Feb 9, 2018 •

edited

TBBle commented Nov 5, 2020 •

edited