Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for subdirectories when building from git source #1684

Closed
thaJeztah opened this issue Sep 14, 2020 · 7 comments · Fixed by #2116
Closed

Add support for subdirectories when building from git source #1684

thaJeztah opened this issue Sep 14, 2020 · 7 comments · Fixed by #2116
Labels
area/feature-parity Feature parity with classic builder help wanted

Comments

@thaJeztah
Copy link
Member

thaJeztah commented Sep 14, 2020

As noted in docker/build-push-action#120 (comment), specifying a sub-directory when building from a git source is not yet supported in BuildKit;

if repo.Subdir != "" {
return nil, errors.Errorf("subdir not supported yet")
}

The PR that added this feature doesn't clearly mention if there's specific reasons for it not to be supported (perhaps there's technical limitations); #63

The classic builder support this https://github.com/docker/cli/blame/2955ece0244307bdbb626ad2471024a51080ff1a/docs/reference/commandline/build.md#L108-L111, so for feature-parity, we should add support in BuildKit as well

Build Syntax Suffix Commit Used Build Context Used
myrepo.git refs/heads/master /
myrepo.git#mytag refs/tags/mytag /
myrepo.git#mybranch refs/heads/mybranch /
myrepo.git#pull/42/head refs/pull/42/head /
myrepo.git#:myfolder refs/heads/master /myfolder
myrepo.git#master:myfolder refs/heads/master /myfolder
myrepo.git#mytag:myfolder refs/tags/mytag /myfolder
myrepo.git#mybranch:myfolder refs/heads/mybranch /myfolder
@tiborvass
Copy link
Collaborator

tiborvass commented Dec 18, 2020

This would be great indeed, but if we want further optimizations in git protocol, I found git clone --filter available since git 2.19:
https://stackoverflow.com/questions/600079/how-do-i-clone-a-subdirectory-only-of-a-git-repository/52269934#52269934

However, in the code we are not using git clone, but git init + git remote + git fetch. The weird thing is that git fetch --filter is not documented (oversight?) but does exist in the code.

Another caveat is that --filter only works with repos that have explicit support for partial clone, greatly limiting the reach of this optimization. Unclear whether there's a tradeoff or why it's not default.

@turbolent
Copy link

turbolent commented Jan 22, 2021

Just ran into this, but was confused, because I wasn't using a subdirectory, I had specified :/.

Due to

if repo.Subdir != "" {
return nil, errors.Errorf("subdir not supported yet")
}
, this is treated as a subdirectory, even though it just indicates the root of the repository. Removing this suffix fixed the problem.

Maybe the subdirectory check should be exteneded to allow "/"?

Hope this helps someone else who runs into this.

@thaJeztah
Copy link
Member Author

thaJeztah commented Jan 23, 2021

Maybe the subdirectory check should be exteneded to allow "/"?

Sounds reasonable to ignore that case (my 0.02c); feel free to open a pull request for that

@kechaoyang01
Copy link

kechaoyang01 commented Feb 22, 2021

any progress for this issues? my apps contain multiple containers and subfolder, need this to be fixed.

@raton465
Copy link

raton465 commented Feb 25, 2021

busco el doker de tcp y udp

@CraigHead
Copy link

CraigHead commented Apr 8, 2021

Second. Any traction on this? We keep our dockerfiles in a ci directory and this currently is blocking us outside of us moving 200 dockerfiles to the root directory of the repo.

Jean1dev added a commit to Jean1dev/cabal-neo-backend that referenced this issue Apr 19, 2021
… you can use the path context in the meantime. More info on Docker docs website.
@justincormack
Copy link
Contributor

justincormack commented May 19, 2021

Sparse checkout now works on github see https://stackoverflow.com/questions/600079/how-do-i-clone-a-subdirectory-only-of-a-git-repository and https://github.blog/2020-01-17-bring-your-monorepo-down-to-size-with-sparse-checkout/ - not sure where else though, it is not clear if we should have a fallback or fail if the server does not support it.

flavray pushed a commit to flavray/docker-postgis that referenced this issue Jul 15, 2021
docker build does not seem to like finding Dockerfiles in
sub-directories.

moby/buildkit#1684
flavray pushed a commit to flavray/docker-postgis that referenced this issue Jul 15, 2021
docker build does not seem to like finding Dockerfiles in
sub-directories.

moby/buildkit#1684
flavray pushed a commit to flavray/docker-postgis that referenced this issue Jul 15, 2021
docker build does not seem to like finding Dockerfiles in
sub-directories.

moby/buildkit#1684
flavray pushed a commit to flavray/docker-postgis that referenced this issue Jul 15, 2021
docker build does not seem to like finding Dockerfiles in
sub-directories.

moby/buildkit#1684
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/feature-parity Feature parity with classic builder help wanted
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants
@turbolent @justincormack @tiborvass @thaJeztah @CraigHead @kechaoyang01 @raton465 and others