Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

staged-dockerfile: support Dockerfile builder with distributed builds #2215

Open
distorhead opened this issue Mar 10, 2020 · 6 comments
Open

Comments

@distorhead
Copy link
Member

distorhead commented Mar 10, 2020

Dockerfile builder should map dockerfile instructions onto stapel build stages. Stapel build stages will be stored in the stages storage.

@distorhead distorhead changed the title Support Dockerfile builder with distributed builds staged-dockerfile: support Dockerfile builder with distributed builds Sep 27, 2022
@distorhead
Copy link
Member Author

distorhead commented Sep 27, 2022

Implement container backend support for dockerfile

  • Use driver pkg/buildah.
  • Implement BuildDockerfileStage
    • no Dockerfile []byte param, no parsing.
    • pass all params using options.
    • map all Dockerfile instructions into options.

Cleanup buildah driver

  • remove docker-with-fuse obsolete mode

@distorhead
Copy link
Member Author

distorhead commented Sep 27, 2022

Refactor mapping of werf.yaml to image+stages

  • Move image+stages initialization into separate package.
  • Define interface to convert parsed werf.yaml or Dockerfile to image+stages.
  • Do not implement Dockerfile to image+stages mapping yet.

@distorhead
Copy link
Member Author

distorhead commented Oct 3, 2022

Refactor mapping of werf.yaml to image+stages

  • Move image+stages initialization into separate package.
  • Define interface to convert parsed werf.yaml or Dockerfile to image+stages.
  • Do not implement Dockerfile to image+stages mapping yet.

Done in #4973.

@distorhead
Copy link
Member Author

distorhead commented Oct 3, 2022

Next step is to implement 3 parts of new dockerfile builder:

  1. Convert config.WerfConfig to dockerfile.Dockerfile object. dockerfile.Dockerfile is parsed stages tree, all data from dockerfile should be accessed only using this struct.
  2. Construct images+stages (ImagesSets) from dockerfile.Dockerfile to use it in werf conveyor which builds images in parallel.
  3. Implement stage.DockerfileInstruction new stage object:
    • Calculate digest.
    • Configure builder for backend.
    • Add some unit tests for this stage.
    • Implement dependencies between werf.yaml images.

distorhead added a commit that referenced this issue Oct 10, 2022
… staged dockerfile builder

* Implemented test instruction building using new staged-dockerfile building method of container backend.
* Refactored conveyor so that new builder now correctly coupled to the existing building mechanics.

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Oct 10, 2022
… staged dockerfile builder

* Implemented test instruction building using new staged-dockerfile building method of container backend.
* Refactored conveyor so that new builder now correctly coupled to the existing building mechanics.

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Oct 10, 2022
… staged dockerfile builder

* Implemented test instruction building using new staged-dockerfile building method of container backend.
* Refactored conveyor so that new builder now correctly coupled to the existing building mechanics.

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
@distorhead
Copy link
Member Author

distorhead commented Oct 10, 2022

Implemented working conveyor + backend build mechanics. No real dockerfile parsing yet. Only RUN instruction implemented at conveyor-stage-level (digests calculation).

Next steps:

  • Implement basic primitives for dockerfile parsing.
  • Implement all stages at conveyor-stage-level.
  • Implement container backend instructions options (for example healthcheck options).

distorhead added a commit that referenced this issue Oct 12, 2022
…o dockerfile parser package

Embed instructions data into container backend instructions.

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Oct 12, 2022
…o dockerfile parser package

Embed instructions data into container backend instructions.

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Oct 14, 2022
…configration section

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Oct 14, 2022
…configration section

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
@distorhead
Copy link
Member Author

distorhead commented Oct 25, 2022

Status update. Implemented full working skeleton of staged dockerfile builder.

Next steps:

  • Buildah container backend should support most of instructions options:
    • ADD
    • CMD
    • COPY
    • ENTRYPOINT
    • ENV
    • EXPOSE
    • HEALTHCHECK
    • LABEL
    • MAINTAINER
    • ONBUILD
    • RUN
      • common mounts support
      • ssh / secret mounts support
    • SHELL
    • STOPSIGNAL
    • USER
    • VOLUME
    • WORKDIR
  • Correct digests calculation and builder configuration for all instructions (full pkg/build/stage/instruction implementation) with tests for digest:
  • dockerfile parser ADD --checksum support
  • basic Dockerfile build-args support
  • basic werf.yaml dependencies directive support
  • WARNING: There is no way to ignore the Dockerfile due to docker limitation when building an image for a compressed context that reads from STDIN.
  • Dependencies + meta-args + FROM instruction support.
  • E2e test for dependencies.
  • Support add_host, network and ssh werf.yaml dockerfile config directive.
  • Support ONBUILD instruction from base image.
  • BUG: chaging of base image in FROM instruction does not cause image rebuilding.
  • Check case when several werf.yaml images reference different targets of the same Dockerfile: common stages should map to the same werf-images in conveyor.
  • Check case when two dockerfile stages has the same common instructions: these should use common cache stages.

distorhead added a commit that referenced this issue Oct 25, 2022
…ions and no duplicates in images-sets

1. Panics occured in Healthcheck and Maintainer instructions because of nil param instead of corresponding backend-instruction.
2. Duplicates was in images-sets when several dockerfile-stage instructions refer to the same stage.

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Oct 25, 2022
…ions and no duplicates in images-sets

1. Panics occured in Healthcheck and Maintainer instructions because of nil param instead of corresponding backend-instruction.
2. Duplicates was in images-sets when several dockerfile-stage instructions refer to the same stage.

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
ilya-lesikov pushed a commit that referenced this issue Oct 26, 2022
…ions and no duplicates in images-sets

1. Panics occured in Healthcheck and Maintainer instructions because of nil param instead of corresponding backend-instruction.
2. Duplicates was in images-sets when several dockerfile-stage instructions refer to the same stage.

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Oct 26, 2022
…pansion

* Refactored stage instructions: save backend instruction type by using generics.
* Implemented generic instruction "expansion" mechanism, which should resolve refs to docker dependency stages to built images refs

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Oct 26, 2022
…pansion

* Refactored stage instructions: save backend instruction type by using generics.
* Implemented generic instruction "expansion" mechanism, which should resolve refs to docker dependency stages to built images refs
* Changed digests in all instructions: prepend field name when calculating field digest.

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Oct 27, 2022
…pansion

* Refactored stage instructions: save backend instruction type by using generics.
* Implemented generic instruction "expansion" mechanism, which should resolve refs to docker dependency stages to built images refs
* Changed digests in all instructions: prepend field name when calculating field digest.

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Nov 7, 2022
Implemented 2-stage build-args expansion:
1. Expand all build-args except dependencies-args on early dockerfile parsing stage.
2. Expand dependencies-args when dependencies images names are available during image conveyor processing.

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Nov 7, 2022
Implemented 2-stage build-args expansion:
1. Expand all build-args except dependencies-args on early dockerfile parsing stage.
2. Expand dependencies-args when dependencies images names are available during image conveyor processing.

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Nov 7, 2022
Implemented 2-stage build-args expansion:
1. Expand all build-args except dependencies-args on early dockerfile parsing stage.
2. Expand dependencies-args when dependencies images names are available during image conveyor processing.

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Nov 7, 2022
Implemented 2-stage build-args expansion:
1. Expand all build-args except dependencies-args on early dockerfile parsing stage.
2. Expand dependencies-args when dependencies images names are available during image conveyor processing.

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Nov 7, 2022
Implemented 2-stage build-args expansion:
1. Expand all build-args except dependencies-args on early dockerfile parsing stage.
2. Expand dependencies-args when dependencies images names are available during image conveyor processing.

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Nov 7, 2022
…lding

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Nov 7, 2022
Implemented 2-stage build-args expansion:
1. Expand all build-args except dependencies-args on early dockerfile parsing stage.
2. Expand dependencies-args when dependencies images names are available during image conveyor processing.

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Nov 7, 2022
…lding

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Nov 15, 2022
refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Nov 16, 2022
refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Nov 16, 2022
…ations)

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Nov 17, 2022
…ations)

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Nov 17, 2022
…ations)

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
distorhead added a commit that referenced this issue Nov 17, 2022
…ations)

refs #2215

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: 🐱‍🏍 In Progress
Status: In Progress
Development

No branches or pull requests

1 participant