`quarkus.openshift.arguments` fails on OCP + native mode #24885

pjgg · 2022-04-12T08:33:11Z

Describe the bug

When I deploy a native application that requires custom java arguments into OCP 4.10 the container crash at creation/start time with the following error:

oc get pods 
app-1-dwfb5    0/1     CreateContainerError   0          20s

oc describe pod app-1-dwfb5 

Events:
  Type     Reason          Age               From               Message
  ----     ------          ----              ----               -------
  Normal   Scheduled       63s               default-scheduler  Successfully assigned ts-fgrltckoih/app-1-dwfb5 to ocp4-10-gl2nw-worker-0-nbgvv
  Normal   AddedInterface  61s               multus             Add eth0 [10.128.2.216/23] from openshift-sdn
  Normal   Pulled          59s               kubelet            Successfully pulled image "image-registry.openshift-image-registry.svc:5000/ts-fgrltckoih/app@sha256:525c030e520217f4e049496e4a4a9d26afa93efae750fe36f7c2f38348254efc" in 2.076644547s
  Warning  Failed          58s               kubelet            Error: container create failed: time="2022-04-12T07:51:38Z" level=error msg="runc create failed: unable to start container process: exec: \"ARG1\": executable file not found in $PATH"
  Normal   Pulled          57s               kubelet            Successfully pulled image "image-registry.openshift-image-registry.svc:5000/ts-fgrltckoih/app@sha256:525c030e520217f4e049496e4a4a9d26afa93efae750fe36f7c2f38348254efc" in 139.839247ms
  Warning  Failed          57s               kubelet            Error: container create failed: time="2022-04-12T07:51:39Z" level=error msg="runc create failed: unable to start container process: exec: \"ARG1\": executable file not found in $PATH"
  Warning  Failed          43s               kubelet            Error: container create failed: time="2022-04-12T07:51:53Z" level=error msg="runc create failed: unable to start container process: exec: \"ARG1\": executable file not found in $PATH"
  Normal   Pulled          43s               kubelet            Successfully pulled image "image-registry.openshift-image-registry.svc:5000/ts-fgrltckoih/app@sha256:525c030e520217f4e049496e4a4a9d26afa93efae750fe36f7c2f38348254efc" in 129.635937ms
  Normal   Pulled          31s               kubelet            Successfully pulled image "image-registry.openshift-image-registry.svc:5000/ts-fgrltckoih/app@sha256:525c030e520217f4e049496e4a4a9d26afa93efae750fe36f7c2f38348254efc" in 138.761777ms
  Warning  Failed          31s               kubelet            Error: container create failed: time="2022-04-12T07:52:05Z" level=error msg="runc create failed: unable to start container process: exec: \"ARG1\": executable file not found in $PATH"
  Normal   Pulled          20s               kubelet            Successfully pulled image "image-registry.openshift-image-registry.svc:5000/ts-fgrltckoih/app@sha256:525c030e520217f4e049496e4a4a9d26afa93efae750fe36f7c2f38348254efc" in 229.794292ms
  Warning  Failed          20s               kubelet            Error: container create failed: time="2022-04-12T07:52:16Z" level=error msg="runc create failed: unable to start container process: exec: \"ARG1\": executable file not found in $PATH"
  Normal   Pulling         9s (x6 over 61s)  kubelet            Pulling image "image-registry.openshift-image-registry.svc:5000/ts-fgrltckoih/app@sha256:525c030e520217f4e049496e4a4a9d26afa93efae750fe36f7c2f38348254efc"
  Normal   Pulled          9s                kubelet            Successfully pulled image "image-registry.openshift-image-registry.svc:5000/ts-fgrltckoih/app@sha256:525c030e520217f4e049496e4a4a9d26afa93efae750fe36f7c2f38348254efc" in 128.577373ms
  Warning  Failed          9s                kubelet            Error: container create failed: time="2022-04-12T07:52:27Z" level=error msg="runc create failed: unable to start container process: exec: \"ARG1\": executable file not found in $PATH"

Appliction.properties:

quarkus.openshift.arguments=ARG1,ARG2

Main class:

@QuarkusMain
public final class Main {
    private static final Logger LOG = Logger.getLogger(Main.class);
    private static final String ARGUMENTS_FROM_MAIN = "Received arguments: ";

    private Main() {

    }

    public static void main(String... args) {
        LOG.info(ARGUMENTS_FROM_MAIN + String.join(",", args));
        Quarkus.run(args);
    }
}

Endpoint

@Path("/args")
public class ArgsResource {
    @Inject
    @CommandLineArguments
    String[] args;

    @GET
    @Produces(MediaType.TEXT_PLAIN)
    public String get() {
        return Stream.of(args).collect(Collectors.joining(","));
    }
}

Works in JVM mode fails in native mode (over OCP)

How to reproduce

Requirements:

Java 11/ 17
Quarkus 2.7.5.Final / 2.7.5.ER2
Openshift 4.10

Reproducer:

git clone git@github.com:quarkus-qe/quarkus-test-suite.git

Module: lifecycle-application

mvn -B -V clean verify -Dnative -Dopenshift -fae -Dall-modules -pl lifecycle-application

Expected behavior

Same as JVM mode

The text was updated successfully, but these errors were encountered:

quarkus-bot · 2022-04-12T08:33:13Z

/cc @geoand, @iocanel

geoand · 2022-04-12T08:36:32Z

Can you please paste what the created Deployment (or DeploymentConfig) looks like?

pjgg · 2022-04-12T08:41:31Z

DeploymentConfig (Native mode):

apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
metadata:
  annotations:
    app.openshift.io/vcs-url: <<unknown>>
    app.quarkus.io/commit-id: a1b72bc4da9e3c3fdce77a153858055098f1c35b
    app.quarkus.io/build-timestamp: 2022-04-12 - 08:27:38 +0000
  labels:
    app.kubernetes.io/name: lifecycle-application
    app.kubernetes.io/version: 1.0.0-SNAPSHOT
    app.openshift.io/runtime: quarkus
  name: lifecycle-application
spec:
  replicas: 1
  selector:
    app.kubernetes.io/version: 1.0.0-SNAPSHOT
    app.kubernetes.io/name: lifecycle-application
  template:
    metadata:
      annotations:
        app.openshift.io/vcs-url: <<unknown>>
        app.quarkus.io/commit-id: a1b72bc4da9e3c3fdce77a153858055098f1c35b
        app.quarkus.io/build-timestamp: 2022-04-12 - 08:27:38 +0000
      labels:
        app.kubernetes.io/name: lifecycle-application
        app.openshift.io/runtime: quarkus
        app.kubernetes.io/version: 1.0.0-SNAPSHOT
    spec:
      containers:
        - args:
            - ARG1
            - ARG2
          env:
            - name: KUBERNETES_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: QUARKUS_OPTS
              value: -Dquarkus.http.host=0.0.0.0
            - name: QUARKUS_HOME
              value: /home/quarkus/
          image: pagonzal/lifecycle-application:1.0.0-SNAPSHOT
          imagePullPolicy: Always
          name: lifecycle-application
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
  triggers:
    - imageChangeParams:
        automatic: true
        containerNames:
          - lifecycle-application
        from:
          kind: ImageStreamTag
          name: lifecycle-application:1.0.0-SNAPSHOT
      type: ImageChange

DeploymentConfig (JVM mode):


apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
metadata:
  annotations:
    app.openshift.io/vcs-url: <<unknown>>
    app.quarkus.io/commit-id: a1b72bc4da9e3c3fdce77a153858055098f1c35b
    app.quarkus.io/build-timestamp: 2022-04-12 - 08:39:40 +0000
  labels:
    app.kubernetes.io/name: lifecycle-application
    app.kubernetes.io/version: 1.0.0-SNAPSHOT
    app.openshift.io/runtime: quarkus
  name: lifecycle-application
spec:
  replicas: 1
  selector:
    app.kubernetes.io/name: lifecycle-application
    app.kubernetes.io/version: 1.0.0-SNAPSHOT
  template:
    metadata:
      annotations:
        app.openshift.io/vcs-url: <<unknown>>
        app.quarkus.io/commit-id: a1b72bc4da9e3c3fdce77a153858055098f1c35b
        app.quarkus.io/build-timestamp: 2022-04-12 - 08:39:40 +0000
      labels:
        app.kubernetes.io/version: 1.0.0-SNAPSHOT
        app.kubernetes.io/name: lifecycle-application
        app.openshift.io/runtime: quarkus
    spec:
      containers:
        - args:
            - ARG1
            - ARG2
          command:
            - java
            - -Dquarkus.http.host=0.0.0.0
            - -Djava.util.logging.manager=org.jboss.logmanager.LogManager
            - -jar
            - /deployments/quarkus-run.jar
          env:
            - name: KUBERNETES_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: JAVA_APP_JAR
              value: /deployments/quarkus-run.jar
          image: pagonzal/lifecycle-application:1.0.0-SNAPSHOT
          imagePullPolicy: Always
          name: lifecycle-application
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
  triggers:
    - imageChangeParams:
        automatic: true
        containerNames:
          - lifecycle-application
        from:
          kind: ImageStreamTag
          name: lifecycle-application:1.0.0-SNAPSHOT
      type: ImageChange

geoand · 2022-04-12T08:45:43Z

Thanks a lot.

One more thing, what does the native DeploymentConfig look like when the no args are supplied?

pjgg · 2022-04-12T08:51:19Z

DeploymentConfig without args (Native mode)

apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
metadata:
  annotations:
    app.openshift.io/vcs-url: <<unknown>>
    app.quarkus.io/commit-id: a1b72bc4da9e3c3fdce77a153858055098f1c35b
    app.quarkus.io/build-timestamp: 2022-04-12 - 08:50:54 +0000
  labels:
    app.kubernetes.io/name: lifecycle-application
    app.kubernetes.io/version: 1.0.0-SNAPSHOT
    app.openshift.io/runtime: quarkus
  name: lifecycle-application
spec:
  replicas: 1
  selector:
    app.kubernetes.io/version: 1.0.0-SNAPSHOT
    app.kubernetes.io/name: lifecycle-application
  template:
    metadata:
      annotations:
        app.openshift.io/vcs-url: <<unknown>>
        app.quarkus.io/commit-id: a1b72bc4da9e3c3fdce77a153858055098f1c35b
        app.quarkus.io/build-timestamp: 2022-04-12 - 08:50:54 +0000
      labels:
        app.openshift.io/runtime: quarkus
        app.kubernetes.io/version: 1.0.0-SNAPSHOT
        app.kubernetes.io/name: lifecycle-application
    spec:
      containers:
        - env:
            - name: KUBERNETES_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: QUARKUS_HOME
              value: /home/quarkus/
            - name: QUARKUS_OPTS
              value: -Dquarkus.http.host=0.0.0.0
          image: pagonzal/lifecycle-application:1.0.0-SNAPSHOT
          imagePullPolicy: Always
          name: lifecycle-application
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
  triggers:
    - imageChangeParams:
        automatic: true
        containerNames:
          - lifecycle-application
        from:
          kind: ImageStreamTag
          name: lifecycle-application:1.0.0-SNAPSHOT
      type: ImageChange

geoand · 2022-04-12T08:54:01Z

Thanks a lot.

@iocanel it seems that args is added to the container spec and there is no command (because it's based into the image I guess).
I assume that the command should be added in this case?

iocanel · 2022-04-12T14:29:42Z

@geoand need to investigate

iocanel · 2022-04-13T12:17:39Z

When we built images using the container-image-openshift extension, we try to identify the base image. When the base image is know we don't populate cmd/args. When the image is not known we do.

In this case, it's not expected work neither for java nor for native. It works on java mode by accident as the default image is not recognized (this is a bug).

So, we need to change the expected behavior, but I am not 100% sure which is the way to go. Our options are the following:

Throw an error when user provides args with no command.
Guess the command when the user passes just the args.
Leave it as is (technically it's not wrong to just pass arguments. I think in this case /bin/sh -c is assumed).

iocanel · 2022-04-13T12:18:08Z

@geoand @Sgitario we have been discussing about commands and args recently. Your thoughts?

Sgitario · 2022-04-13T12:25:50Z

From my point of view, the correct option should be the third as when no command is added, it should default to the one from the base image. If the base image does not default an entrypoint, then the failure is expected. (The same reasoning is for JVM).

geoand · 2022-04-14T04:11:51Z

The problem with that is that it makes things very inconsistent and users can't really anticipate what the behavior is

Sgitario · 2022-04-14T04:31:01Z

Yes, sometimes generate the command and others do not, it's not very user friendly.
We should either (1) always generate the command (in k8s and ocp), so we will always ignore the entrypoint from the docker or (2) never generate the command.

iocanel · 2022-04-14T10:02:25Z

My proposal, is to never generate the command unless explicitly specified. This will allow us to be more predictable without excluding the 3rd party images that may not set the entropoint.

This can be done using a property like: quarkus.kubernetes.include-command-and-args.

iocanel · 2022-04-26T13:55:36Z

After sleeping enough on it, I am leaning towards on never generating the command.

michalvavrik · 2022-05-10T16:52:53Z

I also experienced this inconsistence between java and native mode and IHMO the conclusion you settled on should be documented. If you just add -Dnative the app breaks.

mvn io.quarkus.platform:quarkus-maven-plugin:2.8.3.Final:create -DprojectGroupId=org.acme -DprojectArtifactId=oc -Dextensions=quarkus-openshift
cd oc
mvn clean install -Dquarkus.native.container-runtime=docker -Dquarkus.native.container-build=true -Dquarkus.openshift.route.expose=true -Dquarkus.kubernetes.deploy=true -Dquarkus.kubernetes-client.trust-certs=true -Dquarkus.openshift.arguments=A,B -Dnative 
# BUILD SUCCESS, but pod status is 'CreateContainerError'

If user provides his own quarkus.native.builder-image than he is responsible for defining entrypoint, however if quarkus.native.builder-image is in default value, it's not user's fault that we didn't define entrypoint.

I'd like to set command by default if quarkus.native.builder-image is missing and quarkus.openshift.arguments are set and quarkus.openshift.command is blank. Anyway, if you settled against it, please at least allow me to add comment to quarkus.openshift.arguments documentation like: If the value is set and quarkus.native.builder-image is not specified, please set quarkus.openshift.command=/home/quarkus/application in order to make application work in native mode.

michalvavrik · 2022-05-10T17:12:39Z

In this case, it's not expected work neither for java nor for native. It works on java mode by accident as the default image is not recognized (this is a bug).

I'd like to understand this sentence better, please. Does you assume that if user doesn't define quarkus.native.builder-image and define arguments, he should investigate ubi-quarkus-native-image for correct entrypoint? Maybe I mixed things up.

iocanel · 2022-06-01T12:39:50Z

@michalvavrik: Here's how I understand things.

How kubernetes manifests work

If manifests contain neither commands nor args, then the default entrypoint of the image is used.
If it contains just commands (no entrypoint) it is used as entrypoint
if it contains just arguments (no entrypoint) it is used as entrypoint
If contains arguments and image has entrypoint the entrypoint is used as the missing command.

Source: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#define-a-command-and-arguments-when-you-create-a-pod

The last two practically mean that there is no point in having a manifest just containing args. So, I think that under no circumstantces should we prodcue such manifest. Clearly this is not the case as this issue is such an example.

Why does it happen just in native?

Because in some cases, based on the builder-image we use we might set default values for command & arguments. This behavior can mask the issue above, becuase the end manifest may contain the default command and the user provided arugments. This is what happens in jvm mode.

A proper solution

The fact that the current behavior is pretty hard to even explain without a flow chart, is a clear indication that we are wrong and that this part is overengineered. So, IMHO the proper solution should be to remove conditional logic.

Suggested change: Never ever set default values for command / arguments

This means that:

When the user does not specify anything, kubernetes will just call the image entrypoint
When any of the two is provided pass it through to the resulting manifest and let kubernetes do what is described in the top of this comment.

@michalvavrik: Tried to provide a summary. Hope it explains things better.

pjgg added the kind/bug Something isn't working label Apr 12, 2022

quarkus-bot bot added the area/kubernetes label Apr 12, 2022

iocanel self-assigned this Apr 12, 2022

Sgitario mentioned this issue Apr 27, 2022

Property quarkus.openshift.arguments fails in native mode #25196

Closed

iocanel added a commit to iocanel/quarkus that referenced this issue Sep 5, 2022

fix (quarkusio#24885) Do not infer command/args based on s2i image used

94ae416

iocanel added a commit to iocanel/quarkus that referenced this issue Sep 5, 2022

fix (quarkusio#24885) Do not infer command/args based on s2i image used

4c213c8

iocanel mentioned this issue Sep 12, 2022

Fix inconsistency in kubernetes commands / arguments handling between jvm and native #27733

Merged

iocanel added a commit to iocanel/quarkus that referenced this issue Sep 12, 2022

fix (quarkusio#24885) Do not infer command/args based on s2i image used

2642001

iocanel closed this as completed in #27733 Sep 12, 2022

quarkus-bot bot added this to the 2.13 - main milestone Sep 12, 2022

iocanel mentioned this issue Dec 7, 2022

Generate the command for default base image that requires it #29724

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`quarkus.openshift.arguments` fails on OCP + native mode #24885

`quarkus.openshift.arguments` fails on OCP + native mode #24885

pjgg commented Apr 12, 2022 •

edited

quarkus-bot bot commented Apr 12, 2022

geoand commented Apr 12, 2022

pjgg commented Apr 12, 2022

geoand commented Apr 12, 2022

pjgg commented Apr 12, 2022

geoand commented Apr 12, 2022

iocanel commented Apr 12, 2022

iocanel commented Apr 13, 2022

iocanel commented Apr 13, 2022

Sgitario commented Apr 13, 2022

geoand commented Apr 14, 2022

Sgitario commented Apr 14, 2022

iocanel commented Apr 14, 2022

iocanel commented Apr 26, 2022

michalvavrik commented May 10, 2022 •

edited

michalvavrik commented May 10, 2022 •

edited

iocanel commented Jun 1, 2022 •

edited

quarkus.openshift.arguments fails on OCP + native mode #24885

quarkus.openshift.arguments fails on OCP + native mode #24885

Comments

pjgg commented Apr 12, 2022 • edited

Describe the bug

How to reproduce

Expected behavior

quarkus-bot bot commented Apr 12, 2022

geoand commented Apr 12, 2022

pjgg commented Apr 12, 2022

geoand commented Apr 12, 2022

pjgg commented Apr 12, 2022

geoand commented Apr 12, 2022

iocanel commented Apr 12, 2022

iocanel commented Apr 13, 2022

iocanel commented Apr 13, 2022

Sgitario commented Apr 13, 2022

geoand commented Apr 14, 2022

Sgitario commented Apr 14, 2022

iocanel commented Apr 14, 2022

iocanel commented Apr 26, 2022

michalvavrik commented May 10, 2022 • edited

michalvavrik commented May 10, 2022 • edited

iocanel commented Jun 1, 2022 • edited

How kubernetes manifests work

Why does it happen just in native?

A proper solution

Suggested change: Never ever set default values for command / arguments

`quarkus.openshift.arguments` fails on OCP + native mode #24885

`quarkus.openshift.arguments` fails on OCP + native mode #24885

pjgg commented Apr 12, 2022 •

edited

michalvavrik commented May 10, 2022 •

edited

michalvavrik commented May 10, 2022 •

edited

iocanel commented Jun 1, 2022 •

edited