Can't build container via git url in version 18.03.x

[alexxkn]

Description
Can't build container via git url

Steps to reproduce the issue:
Just for example you can try to start building this image

1. docker build https://github.com/docker-library/rabbitmq.git#:/3.7/alpine

Describe the results you received:

unable to prepare context: unable to 'git clone' to temporary context directory: error initializing submodules: usage: git submodule [--quiet] add [-b <branch>] [-f|--force] [--name <name>] [--reference <repository>] [--] <repository> [<path>]

Describe the results you expected:
Successfully built

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:20:16 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:23:58 2018
  OS/Arch:      linux/amd64
  Experimental: false

same thing in
Docker version 18.03.0-ce, build 0520e24

Output of docker info:

Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.16.5-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.853GiB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

[alexxkn]

With the standard version of git, the image is not compiled.

# git version
git version 1.8.3.1

# docker build https://github.com/docker-library/rabbitmq.git#:/3.7/alpine
unable to prepare context: unable to 'git clone' to temporary context directory: error initializing submodules: usage: git submodule [--quiet] add [-b <branch>] [-f|--force] [--name <name>] [--reference <repository>] [--] <repository> [<path>]
   or: git submodule [--quiet] status [--cached] [--recursive] [--] [<path>...]
   or: git submodule [--quiet] init [--] [<path>...]
   or: git submodule [--quiet] deinit [-f|--force] [--] <path>...
   or: git submodule [--quiet] update [--init] [--remote] [-N|--no-fetch] [-f|--force] [--rebase] [--reference <repository>] [--merge] [--recursive] [--] [<path>...]
   or: git submodule [--quiet] summary [--cached|--files] [--summary-limit <n>] [commit] [--] [<path>...]
   or: git submodule [--quiet] foreach [--recursive] <command>
   or: git submodule [--quiet] sync [--recursive] [--] [<path>...]
: exit status 1

But if the package is updated from third-party repository, then the build is working correctly.

# git version
git version 2.15.0
# docker build https://github.com/docker-library/rabbitmq.git#:/3.7/alpine
Sending build context to Docker daemon  18.94kB
Step 1/20 : FROM alpine:3.7
 ---> 3fd9065eaf02
.....

[alexxkn]

But with docker-ce version 17.12.x all working fine

# docker --version
Docker version 17.12.0-ce, build c97c6d6
# git version
git version 1.8.3.1
# docker build https://github.com/docker-library/rabbitmq.git#:/3.7/alpine
Sending build context to Docker daemon  18.94kB
Step 1/20 : FROM alpine:3.7
 ---> 3fd9065eaf02
....

[thaJeztah]

Suspecting this change could be the culprit #35737

ping @tonistiigi

[ttimasdf]

I searched from Google leading me to this page. I tell what I find, for reference.

Docker 18.02 run a git submodule update --init --recursive --depth=1 after a git clone. However --depth parameter is not supported by git below 2.10. My environment, CentOS Linux release 7.5.1804 have git 1.8.3.1 apparently isn't supported.
In a word, no need to downgrade docker, just upgrade your git 😄

# for EL7
sudo yum install https://centos7.iuscommunity.org/ius-release.rpm
sudo yum swap git git2u

[alexxkn]

It is not always possible to update packages from non-official repos. This method can not be considered a solution to this problem, rather as a workaround.

[thaJeztah]

Opened #37469, but I'm not sure if that's the best approach

[david-l-riley]

I like that approach. Hope it gets merged soon, but I don't know how soon it would make it into mainstream Docker after that. Is there a workaround without doing piles of weird shell magic or shoehorning a non-official Git package into our production build environment?

[kinnarr]

The --depth=1 also leads to build failures, when submodules are not at the head of a branch or recent tag.

error: Server does not allow request for unadvertised object 35f510545d55a831372d3113747bf1314ff4f2ef
Fetched in submodule path 'lib/nitro/tests/Catch', but it did not contain 35f510545d55a831372d3113747bf1314ff4f2ef. Direct fetching of that commit failed.
Failed to recurse into submodule path 'lib/nitro'
: exit status 1

[kriddile]

The --depth=1 also leads to build failures, when submodules are not at the head of a branch or recent tag.

error: Server does not allow request for unadvertised object 35f510545d55a831372d3113747bf1314ff4f2ef
Fetched in submodule path 'lib/nitro/tests/Catch', but it did not contain 35f510545d55a831372d3113747bf1314ff4f2ef. Direct fetching of that commit failed.
Failed to recurse into submodule path 'lib/nitro'
: exit status 1

This is a big problem for us as well, since we can no longer build images directly from a git URL when the git repo contains submodules that are not at the head of a branch or recent tag. Perhaps an optional command line switch to disable the --depth=1 could be added to the build command? Or is there currently a workaround for this problem that I'm not aware of?