Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Project not building since recent update on image #221

Open
ChrisLeeBare opened this issue Aug 26, 2021 · 26 comments
Open

Project not building since recent update on image #221

ChrisLeeBare opened this issue Aug 26, 2021 · 26 comments

Comments

@ChrisLeeBare
Copy link

ChrisLeeBare commented Aug 26, 2021 •
edited
Loading

I am using maven:3-jdk-11 in our gitlab env for building our java applications.
Since the update all pipelines using this image fail with the error:

$ mvn clean package -U
ls: cannot access '/usr/bin/mvn': Operation not permitted

This happens with the image with this sha256 hash:

Using Docker executor with image maven:3-jdk-11 ...
Pulling docker image maven:3-jdk-11 ...
Using docker image sha256:e7518af4d2815035b4ac8ba0e7422c71ae7a1449a38ab5f61be1515c8bc9ac8e for maven:3-jdk-11 ...

The pipelines that ran recently (about 5 days ago) used the following image and they all worked:

Using Docker executor with image maven:3-jdk-11 ...
Pulling docker image maven:3-jdk-11 ...
Using docker image sha256:2232bcac58e0f93a9b5931be9e1be9a270bd27d88dde05fd796f66854851d59c for maven:3-jdk-11 ...

Any help would be appreciated.
@carlossg
Copy link
Owner

works for me

docker run -ti --rm maven:3-jdk-11
maven:3-jdk-11
Unable to find image 'maven:3-jdk-11' locally
3-jdk-11: Pulling from library/maven
4c25b3090c26: Pull complete
1acf565088aa: Pull complete
b95c0dd0dc0d: Pull complete
5cf06daf6561: Pull complete
4e432034bc67: Pull complete
7856767ece4e: Pull complete
c2503997378c: Pull complete
536a90e53c27: Pull complete
bd28ce1d6d6d: Pull complete
ebee2229c385: Pull complete
Digest: sha256:5a7f923651fbff830b702c6423c0a1488576622ca1f16bd6919361d711075623
Status: Downloaded newer image for maven:3-jdk-11
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.071 s
[INFO] Finished at: 2021-08-26T10:43:14Z
[INFO] ------------------------------------------------------------------------

@jpmaas
Copy link

jpmaas commented Aug 26, 2021 •
edited
Loading

we're having the exact same issue with 3-jdk-11-slim. When looking into the container to see the permissions, I get the following:

$ docker run -it maven:3-jdk-11-slim /bin/sh
# ls -la
ls: cannot access '.': Operation not permitted
ls: cannot access '..': Operation not permitted
ls: cannot access 'bin': Operation not permitted
ls: cannot access 'boot': Operation not permitted
ls: cannot access 'dev': Operation not permitted
ls: cannot access 'etc': Operation not permitted
ls: cannot access 'home': Operation not permitted
ls: cannot access 'lib': Operation not permitted
ls: cannot access 'lib64': Operation not permitted
ls: cannot access 'media': Operation not permitted
ls: cannot access 'mnt': Operation not permitted
ls: cannot access 'opt': Operation not permitted
ls: cannot access 'proc': Operation not permitted
ls: cannot access 'root': Operation not permitted
ls: cannot access 'run': Operation not permitted
ls: cannot access 'sbin': Operation not permitted
ls: cannot access 'srv': Operation not permitted
ls: cannot access 'sys': Operation not permitted
ls: cannot access 'tmp': Operation not permitted
ls: cannot access 'usr': Operation not permitted
ls: cannot access 'var': Operation not permitted
ls: cannot access '.dockerenv': Operation not permitted
total 0
d????????? ? ? ? ?            ? .
d????????? ? ? ? ?            ? ..
-????????? ? ? ? ?            ? .dockerenv
d????????? ? ? ? ?            ? bin
d????????? ? ? ? ?            ? boot
d????????? ? ? ? ?            ? dev
d????????? ? ? ? ?            ? etc
d????????? ? ? ? ?            ? home
d????????? ? ? ? ?            ? lib
d????????? ? ? ? ?            ? lib64
d????????? ? ? ? ?            ? media
d????????? ? ? ? ?            ? mnt
d????????? ? ? ? ?            ? opt
d????????? ? ? ? ?            ? proc
d????????? ? ? ? ?            ? root
d????????? ? ? ? ?            ? run
d????????? ? ? ? ?            ? sbin
d????????? ? ? ? ?            ? srv
d????????? ? ? ? ?            ? sys
d????????? ? ? ? ?            ? tmp
d????????? ? ? ? ?            ? usr
d????????? ? ? ? ?            ? var
# whoami
root

SHA: maven@sha256:c48d52ceadf4d11634ffc9e291b8eedc417c52a104a375d57aee1d4c365376ae

As a fallback, we have reverted to using maven:3.8.1-jdk-11-slim, which works fine

@ChrisLeeBare
Copy link
Author

Can confirm reverting the docker image in one of our pipeline to maven:3.8.1-jdk-11 "fixes" the problem.

Using Docker executor with image maven:3.8.1-jdk-11 ...
Pulling docker image maven:3.8.1-jdk-11 ...
Using docker image sha256:5b508b1fe19e290255c9e077a1c7af028a576cabb70eab4abdfee574599f729f for maven:3.8.1-jdk-11
$ mvn clean package -U
[INFO] Scanning for projects...
....
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  33.795 s
[INFO] Finished at: 2021-08-26T11:44:14Z
[INFO] ------------------------------------------------------------------------

All other pipelines with maven:3.8.2-jdk-11 still failing with the exact same issue.

@mawx
Copy link

mawx commented Aug 26, 2021

I confirm I do have the same problem (ls: cannot access 'usr': Operation not permitted) on centos environment for image maven:3-jdk-11
The error doesn't seem to occur on my local (macos environment)

@carlossg
Copy link
Owner

please provide the output of

docker version
docker pull maven:3-jdk-11
docker run --rm maven@sha256:5a7f923651fbff830b702c6423c0a1488576622ca1f16bd6919361d711075623 mvn -v
docker pull maven:3-jdk-11-slim
docker run --rm maven@sha256:c48d52ceadf4d11634ffc9e291b8eedc417c52a104a375d57aee1d4c365376ae mvn -v

@daviddob
Copy link

Can confirm we are seeing similar issues as of a few hours ago as well with both maven:3-jdk-8 and maven:3-jdk-8-slim

Successfully pulled maven@sha256:671806ef232960000b1fbfb0514b4f95ad93318ce525e0d6a7593a5384d86108.

selected worker: c2660d7d-520f-4100-879e-f258de47aec0
running mvn --version
ls: cannot access '/usr/bin/mvn': Operation not permitted
Error: Could not find or load main class org.codehaus.plexus.classworlds.launcher.Launcher
failed

@sayhitoarvind

This comment has been minimized.

Sign in to view
@RafaelGruhn

This comment has been minimized.

Sign in to view
@ChrisLeeBare
Copy link
Author 

Client: Docker Engine - Community
 Version:           19.03.1
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        74b1e89
 Built:             Thu Jul 25 21:21:07 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.1
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       74b1e89
  Built:            Thu Jul 25 21:19:36 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683


[root@linux ~]# docker pull maven:3-jdk-11
3-jdk-11: Pulling from library/maven
Digest: sha256:5a7f923651fbff830b702c6423c0a1488576622ca1f16bd6919361d711075623
Status: Image is up to date for maven:3-jdk-11
docker.io/library/maven:3-jdk-11

[root@linux ~]# docker run --rm maven@sha256:5a7f923651fbff830b702c6423c0a1488576622ca1f16bd6919361d711075623 mvn -v
ls: cannot access '/usr/bin/mvn': Operation not permitted
Error: Could not find or load main class org.codehaus.plexus.classworlds.launcher.Launcher
Caused by: java.lang.ClassNotFoundException: org.codehaus.plexus.classworlds.launcher.Launcher

@carlossg
Copy link
Owner

Looks like a linux only issue with the images built by Docker inc.
Can you try the csanchez/maven and ghcr.io/carlossg/maven ?
If that also fails then it is a problem with the docker engine, not with the images

Also it would be good to know if the parent images have issues

docker run -ti --rm openjdk:11-jdk java -version
docker run -ti --rm openjdk:11-jdk ls -laF /

@ChrisLeeBare
Copy link
Author

The parent images seem to work.
Here's an excerpt from my .gitlab-ci.yml

image: csanchez/maven

The console output:

Using Docker executor with image csanchez/maven ...
00:48
Pulling docker image csanchez/maven ...
Using docker image sha256:5d0e7f67db83ac0ad2dae918cc3f9c8525be240cd02a232f59c0ef1125f6b082 for csanchez/maven ...
Running on runner-198cd238-project-60-concurrent-0 via linux...

Build successful:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  32.940 s
[INFO] Finished at: 2021-08-27T05:18:59Z
[INFO] ------------------------------------------------------------------------

Here's the output of your requested docker run:

[root@linux ~]# docker run -ti --rm openjdk:11-jdk java -version
Unable to find image 'openjdk:11-jdk' locally
11-jdk: Pulling from library/openjdk
4c25b3090c26: Already exists
1acf565088aa: Already exists
b95c0dd0dc0d: Already exists
5cf06daf6561: Already exists
4e432034bc67: Already exists
7856767ece4e: Already exists
c2503997378c: Already exists
Digest: sha256:29c4d68ce0a172ecb915360e3012c73212a8365a7dde63c7d4fb6428021dce4e
Status: Downloaded newer image for openjdk:11-jdk
openjdk version "11.0.12" 2021-07-20
OpenJDK Runtime Environment 18.9 (build 11.0.12+7)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.12+7, mixed mode, sharing)

[root@linux ~]# docker run -ti --rm openjdk:11-jdk ls -laF /
ls: cannot access '/': Operation not permitted

@vmaks

This comment has been minimized.

Sign in to view
@carlossg
Copy link
Owner

The parent images are the openjdk ones and looks like those are not working on linux
My guess is that they were updated and then maven ones rebuilt from them, breaking
In the meantime you can use csanchez/maven or ghcr.io/carlossg/maven

Please abstain to create me too comments if no new information is provided, and just click the thumbs up

@carlossg
Copy link
Owner

in ubuntu focal with the both docker 19 and 20 it works. What OS are you running?

$ docker version
Client:
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.8
 Git commit:        20.10.7-0ubuntu1~20.04.1
 Built:             Wed Aug  4 22:52:25 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Server:
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       20.10.7-0ubuntu1~20.04.1
  Built:            Wed Aug  4 19:07:47 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.2-0ubuntu1~20.04.2
  GitCommit:        
 runc:
  Version:          1.0.0~rc95-0ubuntu1~20.04.2
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        

$ docker run --rm maven@sha256:5a7f923651fbff830b702c6423c0a1488576622ca1f16bd6919361d711075623 mvn -v
Apache Maven 3.8.2 (ea98e05a04480131370aa0c110b8c54cf726c06f)
Maven home: /usr/share/maven
Java version: 11.0.12, vendor: Oracle Corporation, runtime: /usr/local/openjdk-11
Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "5.11.0-1017-gcp", arch: "amd64", family: "unix"

docker version
Client:
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.13.8
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 23:42:35 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       afacb8b7f0
  Built:            Wed Mar 11 22:48:33 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu2
  GitCommit:        
 runc:
  Version:          1.0.0~rc95-0ubuntu1~20.04.2
  GitCommit:        
 docker-init:
  Version:          0.18.0
  GitCommit:        
carlossg@docker:~$ docker run --rm maven@sha256:5a7f923651fbff830b702c6423c0a1488576622ca1f16bd6919361d711075623 mvn -v
Apache Maven 3.8.2 (ea98e05a04480131370aa0c110b8c54cf726c06f)
Maven home: /usr/share/maven
Java version: 11.0.12, vendor: Oracle Corporation, runtime: /usr/local/openjdk-11
Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "5.11.0-1017-gcp", arch: "amd64", family: "unix"

@ChrisLeeBare
Copy link
Author 

[root@linux ~]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

@robert2411
Copy link

I have the same issue on my side and I have the same OS as @ChrisLeeBare so I guess its linked to the centOs family

@jpmaas
Copy link

jpmaas commented Aug 27, 2021

Same here, also CentOS 7.

Another thing to add here is that it definitely seems to be related to the used openjdk distribution because running the build with the version 3-adoptopenjdk-11 does work. I have opened an issue for the openjdk project, see docker-library/openjdk#465.

@carlossg
Copy link
Owner

I tried CentOS 7 with your same docker and containerd version and it works

$ docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:27:04 2020
 OS/Arch:           linux/amd64
 Experimental:      false
Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:25:42 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
$ docker run --rm maven@sha256:5a7f923651fbff830b702c6423c0a1488576622ca1f16bd6919361d71
1075623 mvn -v
Apache Maven 3.8.2 (ea98e05a04480131370aa0c110b8c54cf726c06f)
Maven home: /usr/share/maven
Java version: 11.0.12, vendor: Oracle Corporation, runtime: /usr/local/openjdk-11
Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-1160.36.2.el7.x86_64", arch: "amd64", family: "unix"

@robert2411
Copy link

My output i hope it helps

[]$  cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[]$ docker version
Client: Docker Engine - Community
 Version:           19.03.11
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        42e35e61f3
 Built:             Mon Jun  1 09:13:48 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:45:28 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

[]$ docker pull maven:3-jdk-11
3-jdk-11: Pulling from library/maven
Digest: sha256:5a7f923651fbff830b702c6423c0a1488576622ca1f16bd6919361d711075623
Status: Image is up to date for maven:3-jdk-11
docker.io/library/maven:3-jdk-11

[]$ docker run --rm maven:3-jdk-11 mvn -v
ls: cannot access '/usr/bin/mvn': Operation not permitted
Error: Could not find or load main class org.codehaus.plexus.classworlds.launcher.Launcher
Caused by: java.lang.ClassNotFoundException: org.codehaus.plexus.classworlds.launcher.Launcher

@carlossg
Copy link
Owner

see docker-library/openjdk#465 (comment)

This is likely due to the update to Debian Bullseye -- you'll need to update Docker, runc, and likely libseccomp on your host.
Alternatively, you can pin to the Buster variants explicitly, but you'll want to deal with this properly sooner or later because the Buster variants won't be maintained forever (openjdk:11-jdk-slim-buster).

@clintropolis clintropolis mentioned this issue Aug 29, 2021
Merged
@ChrisLeeBare
Copy link
Author

I updated to docker 20.10.8 and pipelines are running again.

Thanks for the hint.

@risdenk risdenk mentioned this issue Sep 8, 2021
Merged
@shwetha-manvinkurke shwetha-manvinkurke mentioned this issue Sep 8, 2021
Merged
@pjoshi0788
Copy link

Is there any update on the fix?
@carlossg

@oolongtea
Copy link

as a short-term fix, we pinned to image maven:3.6.3-jdk-8

[2021-09-14T17:16:08.015Z] + mvn clean package
[2021-09-14T17:16:08.015Z] ls: cannot access '/usr/bin/mvn': Operation not permitted
[2021-09-14T17:16:08.273Z] Error: Could not find or load main class org.codehaus.plexus.classworlds.launcher.Launcher
script returned exit code 1

@dylan-manchester
Copy link

Can confirm that changing from maven:3.8.3-openjdk-8 to maven:3.6.3-openjdk-8 resolved this issue

@oliviascarfone oliviascarfone mentioned this issue Nov 25, 2021
Merged
@marcinkunert
Copy link

I had the same problem with maven:3.8.4-openjdk-17:

ls: cannot access '/usr/bin/mvn': Operation not permitted
Error: Could not find or load main class org.codehaus.plexus.classworlds.launcher.Launcher
Caused by: java.lang.ClassNotFoundException: org.codehaus.plexus.classworlds.launcher.Launcher
The command '/bin/sh -c mvn dependency:go-offline -B' returned a non-zero code: 1

This issue is not present in maven:3.6.3-openjdk-17

@carlossg
Copy link
Owner

carlossg commented Dec 2, 2021

the fix is commented above, you need to upgrade Docker
#221 (comment)

@carlossg carlossg mentioned this issue Dec 2, 2021
Closed
@carlossg carlossg mentioned this issue Apr 5, 2022
Open
@sferich888 sferich888 mentioned this issue May 17, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests