Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Building sample OS fails #274

Closed
gitlawr opened this issue Jun 15, 2021 · 3 comments
Closed

Building sample OS fails #274

gitlawr opened this issue Jun 15, 2021 · 3 comments
Labels
kind/bug Something isn't working
Projects
Releases

Comments

@gitlawr
Copy link

gitlawr commented Jun 15, 2021

Describe the bug
I run the quick start sample os build and it fails.

cos-toolkit version:
master of the cos-toolkit-sample-repo(cc67643266c80f246e2cbc119170c6b8366a6fa8)

CPU architecture, OS, and Version:

# docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 5
  Running: 1
  Paused: 0
  Stopped: 4
 Images: 99
 Server Version: 20.10.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-1048-aws
 Operating System: Ubuntu 20.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.786GiB
 Name: ip-172-31-15-143
 ID: IA5G:4Y63:4BCK:56BU:SUL4:GWDU:E2OT:LQAF:IDZW:NDI5:ZZHY:FWGD
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

To Reproduce

  1. Provision an ubuntu20.04 EC2 instance
  2. Install docker
  3. Run the quick start build.
$ git clone https://github.com/rancher-sandbox/cos-toolkit-sample-repo
$ cd cos-toolkit-sample-repo
$ source .envrc
$ cos-build

Expected behavior
Build succeeds.

Logs

Step 12/13 : RUN checkmodule -M -m -o cOS.mod cOS.te && semodule_package -o cOS.pp -m cOS.mod
  ---> Using cache
 ---> 1be520969ead
Step 13/13 : RUN semodule -i cOS.pp
  ---> Running in c5bfa5ae92e2
 libsemanage.semanage_commit_sandbox: Error while renaming /var/lib/selinux/targeted/active to /var/lib/selinux/targeted/previous. (Invalid cross-device link).
semodule:  Failed!
 The command '/bin/sh -c semodule -i cOS.pp' returned a non-zero code: 1
 Error: while resolving join images: failed building join image: Failed compiling system/selinux-policies-0.0.6+3: failed building package image: Could not push image: raccos/sampleos:ffc8618ecbfbffc11cc3bca301cc49867eb7dccb623f951dd92caa10ced29b68 selinux-policies-system-0.0.6+3.dockerfile: Could not build image: raccos/sampleos:ffc8618ecbfbffc11cc3bca301cc49867eb7dccb623f951dd92caa10ced29b68 selinux-policies-system-0.0.6+3.dockerfile: Failed running command: : exit status 1
 Bailing out
make: *** [Makefile:45: build] Error 1

Additional context

Possibly related to docker/for-linux#480 but the workaround there does not help.
The metacopy parameter is disabled.

$ cat /sys/module/overlay/parameters/metacopy
N

As the workaround, I change the docker storage driver from overlay2 to devicemapper then the build succeeds.
@gitlawr gitlawr added the kind/bug Something isn't working label Jun 15, 2021
@gitlawr
Copy link
Author

gitlawr commented Jun 15, 2021

Another failure but this one seems to be occasional. A retry succeeds. Print the log here for reference.

 🐋  Building image raccos/sampleos:cc0aee4ff6c194f920a945c45ebcb487c3e22c5ab40e2634ea70c064dfab206d done
 📦  8/8 system/cos-0.5.3+1 ⤑ 🔨  build system/selinux-policies-0.0.6+3 ✅  Done
 🚀  All dependencies are satisfied, building package requested by the user system/cos-0.5.3+1
 📦  system/cos-0.5.3+1  Using image:  raccos/sampleos:cc0aee4ff6c194f920a945c45ebcb487c3e22c5ab40e2634ea70c064dfab206d
 📦  system/cos-0.5.3+1 🐋  Generating 'builder' image from raccos/sampleos:cc0aee4ff6c194f920a945c45ebcb487c3e22c5ab40e2634ea70c064dfab206d as raccos/sampleos:builder-8533d659df2505a518860bd010b7a8ed with prelude steps
🚧  warning Failed to download 'raccos/sampleos:builder-8533d659df2505a518860bd010b7a8ed'. Will keep going and build the image unless you use --fatal
🚧  warning Failed pulling image: Error response from daemon: manifest for raccos/sampleos:builder-8533d659df2505a518860bd010b7a8ed not found: manifest unknown: manifest unknown
: exit status 1
 🐋  Building image raccos/sampleos:builder-8533d659df2505a518860bd010b7a8ed
 Sending build context to Docker daemon  9.728kB
 Step 1/10 : FROM raccos/sampleos:cc0aee4ff6c194f920a945c45ebcb487c3e22c5ab40e2634ea70c064dfab206d
  ---> f1122e79b17e
Step 2/10 : COPY . /luetbuild
  ---> 4ff3e202951b
 Step 3/10 : WORKDIR /luetbuild
  ---> Running in 7ec571b96c6f
 Removing intermediate container 7ec571b96c6f
  ---> 9e05366f830a
Step 4/10 : ENV PACKAGE_NAME=cos
  ---> Running in 30297dbd21a3
 Removing intermediate container 30297dbd21a3
  ---> 4c4838b629f4
 Step 5/10 : ENV PACKAGE_VERSION=0.5.3+1
  ---> Running in 36361b617252
 Removing intermediate container 36361b617252
  ---> 6ac0d3a2ff9a
Step 6/10 : ENV PACKAGE_CATEGORY=system
  ---> Running in f20c2cf3cf34
 Removing intermediate container f20c2cf3cf34
  ---> a902ff95d273
 Step 7/10 : COPY --from=quay.io/costoolkit/build-cache:f3a333095d9915dc17d7f0f5629a638a7571a01dcf84886b48c7b2e5289a668a /usr/bin/yip /usr/bin/yip
  ---> 42fa00d9c990
 Step 8/10 : COPY --from=quay.io/costoolkit/build-cache:e3bbe48c6d57b93599e592c5540ee4ca7916158461773916ce71ef72f30abdd1 /usr/bin/luet /usr/bin/luet
 e3bbe48c6d57b93599e592c5540ee4ca7916158461773916ce71ef72f30abdd1: Pulling from costoolkit/build-cache
 3599716b36e7:  Already exists
 24a39c0e5d06: Already exists
 4f4fb700ef54: Already exists
 4f4fb700ef54: Already exists
 4f4fb700ef54: Already exists
 378615c429f5: Already exists
 c28da22d3dfd:  Already exists
 ddb4dd5c81b0: Already exists
 92db41c0c9ab: Already exists
 4f4fb700ef54: Already exists
 6e0ca71a6514: Already exists
 47debb886c7d: Already exists
 4f4fb700ef54: Already exists
 4f4fb700ef54: Already exists
 4f4fb700ef54: Already exists
 d0c9d0f8ddb6: Already exists
 e5a48f1f72ad:  Pulling fs layer
 4f4fb700ef54:  Pulling fs layer
 7d603b2e4a37:  Pulling fs layer
 64c4d787e344:  Pulling fs layer
 f8835d2e60d1:  Pulling fs layer
 64c4d787e344:  Waiting
 f8835d2e60d1:  Waiting
 e5a48f1f72ad:  Download complete
 e5a48f1f72ad:  Pull complete
 4f4fb700ef54:  Verifying Checksum
 4f4fb700ef54:  Download complete
 4f4fb700ef54:  Pull complete
 7d603b2e4a37: Verifying Checksum
7d603b2e4a37: Download complete
 64c4d787e344: Verifying Checksum
64c4d787e344: Download complete
 7d603b2e4a37: Pull complete
 64c4d787e344: Pull complete
 f8835d2e60d1:  Verifying Checksum
 f8835d2e60d1:  Download complete
 f8835d2e60d1: Pull complete
 Digest: sha256:9b58bed47ff53f2d6cc517a21449cae686db387d171099a4a3145c8a47e6a1e0
 Status: Downloaded newer image for quay.io/costoolkit/build-cache:e3bbe48c6d57b93599e592c5540ee4ca7916158461773916ce71ef72f30abdd1
 failed to export image: failed to create image: failed to get layer sha256:118537d8997a08750ab1ac3d8e8575e40fe60e8337e02633b0d8a1287117fe78: layer does not exist
 Error: while resolving join images: failed building join image: failed building package image: Could not push image: raccos/sampleos:cc0aee4ff6c194f920a945c45ebcb487c3e22c5ab40e2634ea70c064dfab206d cos-system-0.5.3+1-builder.dockerfile: Could not build image: raccos/sampleos:cc0aee4ff6c194f920a945c45ebcb487c3e22c5ab40e2634ea70c064dfab206d cos-system-0.5.3+1-builder.dockerfile: Failed running command: : exit status 1
 Bailing out
make: *** [Makefile:45: build] Error 1

@mudler
Copy link
Contributor

mudler commented Jun 15, 2021 •
edited 

Step 12/13 : RUN checkmodule -M -m -o cOS.mod cOS.te && semodule_package -o cOS.pp -m cOS.mod
  ---> Using cache
 ---> 1be520969ead
Step 13/13 : RUN semodule -i cOS.pp
  ---> Running in c5bfa5ae92e2
 libsemanage.semanage_commit_sandbox: Error while renaming /var/lib/selinux/targeted/active to /var/lib/selinux/targeted/previous. (Invalid cross-device link).
semodule:  Failed!
 The command '/bin/sh -c semodule -i cOS.pp' returned a non-zero code: 1
 Error: while resolving join images: failed building join image: Failed compiling system/selinux-policies-0.0.6+3: failed building package image: Could not push image: raccos/sampleos:ffc8618ecbfbffc11cc3bca301cc49867eb7dccb623f951dd92caa10ced29b68 selinux-policies-system-0.0.6+3.dockerfile: Could not build image: raccos/sampleos:ffc8618ecbfbffc11cc3bca301cc49867eb7dccb623f951dd92caa10ced29b68 selinux-policies-system-0.0.6+3.dockerfile: Failed running command: : exit status 1
 Bailing out
make: *** [Makefile:45: build] Error 1

Looks like indeed an issue with the underlying storage driver, I think we should at least document the workaround in our docs in a proper section, like "Known issues" and alikes

@mudler
Copy link
Contributor

mudler commented Jun 15, 2021 •
edited

Another failure but this one seems to be occasional. A retry succeeds. Print the log here for reference.

 🐋  Building image raccos/sampleos:cc0aee4ff6c194f920a945c45ebcb487c3e22c5ab40e2634ea70c064dfab206d done
 📦  8/8 system/cos-0.5.3+1 ⤑ 🔨  build system/selinux-policies-0.0.6+3 ✅  Done
 🚀  All dependencies are satisfied, building package requested by the user system/cos-0.5.3+1
 📦  system/cos-0.5.3+1  Using image:  raccos/sampleos:cc0aee4ff6c194f920a945c45ebcb487c3e22c5ab40e2634ea70c064dfab206d
 📦  system/cos-0.5.3+1 🐋  Generating 'builder' image from raccos/sampleos:cc0aee4ff6c194f920a945c45ebcb487c3e22c5ab40e2634ea70c064dfab206d as raccos/sampleos:builder-8533d659df2505a518860bd010b7a8ed with prelude steps
🚧  warning Failed to download 'raccos/sampleos:builder-8533d659df2505a518860bd010b7a8ed'. Will keep going and build the image unless you use --fatal
🚧  warning Failed pulling image: Error response from daemon: manifest for raccos/sampleos:builder-8533d659df2505a518860bd010b7a8ed not found: manifest unknown: manifest unknown
: exit status 1
 🐋  Building image raccos/sampleos:builder-8533d659df2505a518860bd010b7a8ed
 Sending build context to Docker daemon  9.728kB
 Step 1/10 : FROM raccos/sampleos:cc0aee4ff6c194f920a945c45ebcb487c3e22c5ab40e2634ea70c064dfab206d
  ---> f1122e79b17e
Step 2/10 : COPY . /luetbuild
  ---> 4ff3e202951b
 Step 3/10 : WORKDIR /luetbuild
  ---> Running in 7ec571b96c6f
 Removing intermediate container 7ec571b96c6f
  ---> 9e05366f830a
Step 4/10 : ENV PACKAGE_NAME=cos
  ---> Running in 30297dbd21a3
 Removing intermediate container 30297dbd21a3
  ---> 4c4838b629f4
 Step 5/10 : ENV PACKAGE_VERSION=0.5.3+1
  ---> Running in 36361b617252
 Removing intermediate container 36361b617252
  ---> 6ac0d3a2ff9a
Step 6/10 : ENV PACKAGE_CATEGORY=system
  ---> Running in f20c2cf3cf34
 Removing intermediate container f20c2cf3cf34
  ---> a902ff95d273
 Step 7/10 : COPY --from=quay.io/costoolkit/build-cache:f3a333095d9915dc17d7f0f5629a638a7571a01dcf84886b48c7b2e5289a668a /usr/bin/yip /usr/bin/yip
  ---> 42fa00d9c990
 Step 8/10 : COPY --from=quay.io/costoolkit/build-cache:e3bbe48c6d57b93599e592c5540ee4ca7916158461773916ce71ef72f30abdd1 /usr/bin/luet /usr/bin/luet
 e3bbe48c6d57b93599e592c5540ee4ca7916158461773916ce71ef72f30abdd1: Pulling from costoolkit/build-cache
 3599716b36e7:  Already exists
 24a39c0e5d06: Already exists
 4f4fb700ef54: Already exists
 4f4fb700ef54: Already exists
 4f4fb700ef54: Already exists
 378615c429f5: Already exists
 c28da22d3dfd:  Already exists
 ddb4dd5c81b0: Already exists
 92db41c0c9ab: Already exists
 4f4fb700ef54: Already exists
 6e0ca71a6514: Already exists
 47debb886c7d: Already exists
 4f4fb700ef54: Already exists
 4f4fb700ef54: Already exists
 4f4fb700ef54: Already exists
 d0c9d0f8ddb6: Already exists
 e5a48f1f72ad:  Pulling fs layer
 4f4fb700ef54:  Pulling fs layer
 7d603b2e4a37:  Pulling fs layer
 64c4d787e344:  Pulling fs layer
 f8835d2e60d1:  Pulling fs layer
 64c4d787e344:  Waiting
 f8835d2e60d1:  Waiting
 e5a48f1f72ad:  Download complete
 e5a48f1f72ad:  Pull complete
 4f4fb700ef54:  Verifying Checksum
 4f4fb700ef54:  Download complete
 4f4fb700ef54:  Pull complete
 7d603b2e4a37: Verifying Checksum
7d603b2e4a37: Download complete
 64c4d787e344: Verifying Checksum
64c4d787e344: Download complete
 7d603b2e4a37: Pull complete
 64c4d787e344: Pull complete
 f8835d2e60d1:  Verifying Checksum
 f8835d2e60d1:  Download complete
 f8835d2e60d1: Pull complete
 Digest: sha256:9b58bed47ff53f2d6cc517a21449cae686db387d171099a4a3145c8a47e6a1e0
 Status: Downloaded newer image for quay.io/costoolkit/build-cache:e3bbe48c6d57b93599e592c5540ee4ca7916158461773916ce71ef72f30abdd1
 failed to export image: failed to create image: failed to get layer sha256:118537d8997a08750ab1ac3d8e8575e40fe60e8337e02633b0d8a1287117fe78: layer does not exist
 Error: while resolving join images: failed building join image: failed building package image: Could not push image: raccos/sampleos:cc0aee4ff6c194f920a945c45ebcb487c3e22c5ab40e2634ea70c064dfab206d cos-system-0.5.3+1-builder.dockerfile: Could not build image: raccos/sampleos:cc0aee4ff6c194f920a945c45ebcb487c3e22c5ab40e2634ea70c064dfab206d cos-system-0.5.3+1-builder.dockerfile: Failed running command: : exit status 1
 Bailing out
make: *** [Makefile:45: build] Error 1

This seems another issue, can you try by enabling docker buildkit with DOCKER_BUILDKIT=1?

I think it's still related to docker: moby/moby#37965

This was referenced Jun 15, 2021
Closed
Closed
@mudler mudler added this to 💡 Untriaged in Releases Jun 15, 2021
@mudler mudler removed their assignment Jun 15, 2021
mudler added a commit that referenced this issue Jun 25, 2021
@mudler
Add Known issues section to our docs
7918715 
Fixes #274
Fixes #275

Signed-off-by: Ettore Di Giacinto <edigiacinto@suse.com>
mudler added a commit that referenced this issue Jun 25, 2021
@mudler
Add Known issues section to our docs
b680c16 
Fixes #274
Fixes #275

Signed-off-by: Ettore Di Giacinto <edigiacinto@suse.com>
mudler added a commit that referenced this issue Jun 25, 2021
@mudler
Add Known issues section to our docs
047a846 
Fixes #274
Fixes #275

Signed-off-by: Ettore Di Giacinto <edigiacinto@suse.com>
mudler added a commit that referenced this issue Jun 25, 2021
@mudler
Add Known issues section to our docs
b7e2f2c 
Fixes #274
Fixes #275

Signed-off-by: Ettore Di Giacinto <edigiacinto@suse.com>
mudler added a commit that referenced this issue Jun 28, 2021
@mudler
Add Known issues section to our docs
17e2bf5 
Fixes #274
Fixes #275

Signed-off-by: Ettore Di Giacinto <edigiacinto@suse.com>
mudler added a commit that referenced this issue Jun 28, 2021
@mudler
Add Known issues section to our docs
7126a04 
Fixes #274
Fixes #275

Signed-off-by: Ettore Di Giacinto <edigiacinto@suse.com>
mudler added a commit that referenced this issue Jun 28, 2021
@mudler
Add Known issues section to our docs
53b0667 
Fixes #274
Fixes #275

Signed-off-by: Ettore Di Giacinto <edigiacinto@suse.com>
mudler added a commit that referenced this issue Jun 28, 2021
@mudler
Add Known issues section to our docs
d070634 
Fixes #274
Fixes #275

Signed-off-by: Ettore Di Giacinto <edigiacinto@suse.com>
mudler added a commit that referenced this issue Jun 29, 2021
@mudler
Add Known issues section to our docs
5d807d9 
Fixes #274
Fixes #275

Signed-off-by: Ettore Di Giacinto <edigiacinto@suse.com>
@mudler mudler closed this as completed in 39a1419 Jun 29, 2021
Releases automation moved this from 💡 Untriaged to ✅ Done Jun 29, 2021
frelon pushed a commit to frelon/elemental-toolkit that referenced this issue May 12, 2023
@cOS-cibot @Itxaka
[auto] Update cmd docs (rancher#274)
a81c51f 
Co-authored-by: Itxaka <Itxaka@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Something isn't working
Projects
No open projects
Releases
✅ Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mudler @gitlawr