Execute commands during Devfile push

[rajivnathan]

Related to #2470

Description

Devfiles can contain a list of commands that can be run against components in the devfile. odo devfile push should execute the applicable commands.

Example: https://github.com/eclipse/che-devfile-registry/blob/master/devfiles/java-mysql/devfile.yaml

Commands section from the example devfile:

commands:
  -
    name: maven build
    actions:
      -
        type: exec
        component: tools
        command: "mvn clean install"
        workdir: "${CHE_PROJECTS_ROOT}/web-java-spring-petclinic"
  -
    name: run webapp
    actions:
      -
        type: exec
        component: tools
        command: |
          SPRING_DATASOURCE_URL=jdbc:mysql://db/petclinic \
          SPRING_DATASOURCE_USERNAME=petclinic \
          SPRING_DATASOURCE_PASSWORD=petclinic \
          java -jar -Dspring.profiles.active=mysql \
          -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 \
          target/*.jar
        workdir: ${CHE_PROJECTS_ROOT}/web-java-spring-petclinic
  -
    name: prepare database
    actions:
    -
      type: exec
      component: mysql
      command: |
        /opt/rh/rh-mysql57/root/usr/bin/mysql -u root < ${CHE_PROJECTS_ROOT}/web-java-spring-petclinic/src/main/resources/db/mysql/user.sql &&
        /opt/rh/rh-mysql57/root/usr/bin/mysql -u root petclinic < ${CHE_PROJECTS_ROOT}/web-java-spring-petclinic/src/main/resources/db/mysql/schema.sql &&
        echo -e "\e[32mDone.\e[0m Database petclinic was configured!"
  - name: Debug remote java application
    actions:
      - type: vscode-launch
        referenceContent: |
          {
          "version": "0.2.0",
          "configurations": [
            {
              "type": "java",
              "name": "Debug (Attach) - Remote",
              "request": "attach",
              "hostName": "localhost",
              "port": 5005
            }]
          }

Implementation plan

• When a user invokes odo push odo shall look for the devbuild and devrun commands and execute them after the sync step. (These are the commands looked for during create, see Devfile support for odo create #2699)
• Use supervisord as the entrypoint for the container that will be used by the run command, this will ensure the container does not exit and the runtime process can be stopped and started independently using supervisord
• The odo-init-image can be updated to include a supervisor config and script that can be used for devfile support
• By default odo will use supervisord to stop and then start the run command's process when odo push is invoked. However, if there is a top level attribute odo.restart set to false then the run process should not be stopped if it is running.

attributes:
  odo.restart: "false"

Proof of concept

https://github.com/rajivnathan/odo-init-image/tree/master/execInvestigation

Acceptance Criteria

• odo push should execut commands with alias devbuild and devrun commands
• Push and watch should accept command flags to allow users to execute build and run commands other than the default ones eg. odo push --build-command=hotdeploy --run-command=run and odo watch --build-command=hotdeploy --run-command=run
• Only commands for supported component types should be considered runnable (currently just dockerimage components)
• Only copy supervisord into the container that has a corresponding run command
• Use supervisord as the entrypoint by default
• If devfile has an entrypoint specified then do not use supervisord as the entrypoint but initialize it after the container has started and run other commands using supervisord
• Run process should not be restarted if odo.restart: "false" attribute is set.
• odo push should result in an error if the devbuild and devrun commands cannot be found
• odo push should result in an error if the --build-command=<alias> and --run-command=<alias> options were specified but the alias cannot be found

[kind/Feature]

[kadel]

/area devfile
/kind feature

[rajivnathan]

/assign

[kadel]

Continuing the discussion from #2471 (comment)

@kadel That's going to be tricky. I don't think we have a good solution for making it work for existing Devfiles that have commands that use long-running processes (no daemon or fork to background). While doing the POC with IDPs we used nohup for commands that needed to continue to run in the background. This worked when I tested it in Che. When I ran the command in Che without nohup then when I closed the command output in Che (effectively terminating the shell) the command process also terminated but after modifying the script for the command to use nohup then I was able to close the command in Che but leave the server running. But for that to work we would require the stack owner to ensure they do the same but that's not a great experience for stack creators. It would be better if we could do something at the odo level like run the command with nohup to ensure it can continue in the background. I suppose we could try to wrap any command with nohup if it's not already using it but I'm not sure if there would be problems with that approach. Any other ideas?

Agreed, we need to have something on the odo level that solves this. Stack owner should not be responsible for this especially as this is not an easy thing to do, and this would have to be implemented in every stack.

It is not just about running this on the background, odo also needs to be able to control that process (stop and start it).

In the "LocalConfig/S2I" implementation odo is using SupervisorD exactly for this. It is a little bit painful to handle everything and also deal with initContainer, but so far it worked pretty well.

The way how it works, that we use initContainer to inject supervisord and config file into the container.
supervisord process is used as an entry point of the container. The application process is then executed as SupervisorD service (child to a supervisord process). Than using SupervisorD api we can start stop the application process.

As for update, wouldn't it be the stack that should know how to handle updates correctly? In some cases it's not necessary because it's built into the runtime. eg. In the case of liberty stop/start commands can be used and it tracks the PID on its own. In some cases we don't want to restart the process if it isn't required because the runtime supports hot deployment.

Odo needs to know if the restart is needed or not.
I think that by default odo should assume that restart is required, as this is the safest option that works every time. If runtime supports hot redeployment we could use something in Devfile to hint that restart is not needed. We could use top level attributes in devfile 1.0.0 (https://github.com/redhat-developer/devfile/blob/master/docs/devfile.md#attributes).

attributes:
   odo.hotrealod: true

or inverse

attributes:
   odo.restart: false

[rajivnathan]

@maysunfaisal and I have done some investigation on a solution with nohup vs supervisord. We found that a lot of what we'd want to do with nohup is already handled nicely by supervisord. The initial apprehension about using supervisord was around whether it would work with a wide variety of container images but I was able to use it on a plain alpine container which is pretty bare bones so I'm less concerned about that now. I'll update the description of this issue with a plan for implementation using supervisord that can be reviewed by others as well.

[kadel]

• Push and watch should accept command flags to allow users to run commands other than the default ones eg. odo push --command=hotdeploy and odo watch --command=hotdeploy

I guess that this most cases it will be 2 step operation (build, run) so there should probably be 2 falls (- --run-command, --build-command`)

• Use supervisord as the entrypoint by default

• If devfile has an entrypoint specified then do not use supervisord as the entrypoint but initialize it after the container has started and run other commands using supervisord

Not sure that I understand this :-/

• Run process should not be stopped if odo.restart: "false" attribute is set.

+1

Otherwise, this looks good.
Let me know if you need any help, as this is almost the same as what I implemented for "legacy components" in one of the first odo versions.

Maybe dumb idea, but I was also thinking if it would be possible to embed supervisord binary into odo itself, and then somehow inject it into the container, to avoid using initContainer.
I'll try to validate this idea, we could use it in future if it is possible

[rajivnathan]

@kadel

I guess that this most cases it will be 2 step operation (build, run) so there should probably be 2 falls (- --run-command, --build-command`)

So we are not going to worry about executing any other commands in the devfile? ie. only build and run?

Update: We'll focus on build and run commands only for the tech preview since that is what the majority of devfiles require and satisfies the basic requirements for odo support.

Use supervisord as the entrypoint by default

If devfile has an entrypoint specified then do not use supervisord as the entrypoint but initialize it after the container has started and run other commands using supervisord

The first part of that is if the devfile creator specifically set a command for the component then that is meant to be the entrypoint for that container so we should not use go-init as the entrypoint otherwise there is no way for them to override the entrypoint for the container?

The second part of that is if they override the entrypoint, we can still run commands using supervisord by initializing it after the container has come up so that we still get the benefits of process management.

Any ideas on a better way to handle when a devfile writer has specified a command (entrypoint) for their component?

Update: We'll stick with this approach because there isn't a better option for now.

Maybe dumb idea, but I was also thinking if it would be possible to embed supervisord binary into odo itself, and then somehow inject it into the container, to avoid using initContainer.
I'll try to validate this idea, we could use it in future if it is possible

That sounds like a great idea if we can pull it off! I think we'd need to be careful about file permission issues though.

[rajivnathan]

I've tried the execute PR with a few modified devfiles from the che-devfile-registry. Here is the summary:

Devfiles status

Devfile	Status	Notes
go	❌	see notes below
java-gradle	❌	see notes below
java-maven	✅
java-mongo		need to revisit once service PR is in
java-mysql	❌	see notes below
java-web-spring	❌	maven build issues, not related to odo
java-web-vertx	✅
nodejs	✅
python-django	✅	verified api/ endpoint came up
python	✅	No build command
quarkus	❌	devfile problem, see notes below

Notes

General

• devfile references che-specific environment variable ${CHE_PROJECTS_ROOT} for the location where project source is copied/cloned
  • odo should set the ${CHE_PROJECTS_ROOT} environment variable

Usability notes

• Some devfiles don't have a build command
• It's annoying to have to type --build-command and --run-command options for odo push, should be able to persist it so that it's not needed for subsequent uses of odo push
• Need to determine how debug will work, some devfiles have a run server in debug mode command of type exec as well as a debug command of type vscode-launch

java-gradle

Expected Build Output (Taken from using the same Devfile on Che)

Picked up JAVA_TOOL_OPTIONS: -XX:MaxRAMPercentage=50 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xms20m -Djava.security.egd=file:/dev/./urandom

Welcome to Gradle 6.2.1!

Here are the highlights of this release:
 - Dependency checksum and signature verification
 - Shareable read-only dependency cache
 - Documentation links in deprecation messages

For more details see https://docs.gradle.org/6.2.1/release-notes.html

Starting a Gradle Daemon (subsequent builds will be faster)

> Task :test
Picked up JAVA_TOOL_OPTIONS: -XX:MaxRAMPercentage=50 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xms20m -Djava.security.egd=file:/dev/./urandom

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.2.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 22s
7 actionable tasks: 7 executed

Actual Build Output

Picked up JAVA_TOOL_OPTIONS: -XX:MaxRAMPercentage=50 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xms20m -Djava.security.egd=file:/dev/./urandom

Welcome to Gradle 6.2.1!

Here are the highlights of this release:
 - Dependency checksum and signature verification
 - Shareable read-only dependency cache
 - Documentation links in deprecation messages

For more details see https://docs.gradle.org/6.2.1/release-notes.html


FAILURE: Build failed with an exception.

* What went wrong:
Failed to release lock on daemon addresses registry

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org
 ✗  
Unable to build files
Picked up JAVA_TOOL_OPTIONS: -XX:MaxRAMPercentage=50 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xms20m -Djava.security.egd=file:/dev/./urandom

Welcome to Gradle 6.2.1!

Here are the highlights of this release:
 - Dependency checksum and signature verification
 - Shareable read-only dependency cache
 - Documentation links in deprecation messages

For more details see https://docs.gradle.org/6.2.1/release-notes.html


FAILURE: Build failed with an exception.

* What went wrong:
Failed to release lock on daemon addresses registry

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

go

Problematic because the code currently requires both a build and run command but the go devfile doesn't really have a build command, the run command uses a single run command go get -d && go run main.go

• Action: Make build commands optional, it should be possible to use odo push --run-command="run outyet" without any --build-command option specified

 ✓  Waiting for component to start [42s]
I0319 14:58:14.413135   93206 pods.go:54] Pod example-7d95d95f5c-rsphf is Running
I0319 14:58:14.413224   93206 adapter.go:258] Push: componentName: example, path: /Users/rajiv/git/example, files: [], delFiles: [], isForcePush: true
 •  Syncing files to the component  ...
I0319 14:58:14.413495   93206 adapter.go:282] Creating /projects/example on the remote container if it doesn't already exist
I0319 14:58:15.766751   93206 adapter.go:311] Copying files  to pod
 ✓  Syncing files to the component [3s]
I0319 14:58:17.741924   93206 adapter.go:328] mjf  execDevfileCmd pushDevfileCommands [{[{0xc0002f0f30 0xc0002f0f40 <nil> <nil> 0xc0002f0f50 <nil>}] map[] stop outyet {<nil> <nil>}} {[{0xc0002f0ef0 0xc0002f0f00 <nil> <nil> 0xc0002f0f10 0xc0002f0f20}] map[] run outyet {<nil> <nil>}}]
I0319 14:58:17.742607   93206 adapter.go:335] mjf executing command stop outyet
 ✗  Push devfile component example [46s]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x21b66c9]

goroutine 1 [running]:
github.com/openshift/odo/pkg/devfile/adapters/kubernetes/component.Adapter.execDevfileCmd(0x2ab0fe0, 0xc0008efe60, 0x2a69ce0, 0xc00080bf90, 0xc0002beb40, 0xc00077cbf9, 0x3, 0xc000441560, 0xc0007f4108, 0x7, ...)
	/Users/rajiv/go/src/github.com/openshift/odo/pkg/devfile/adapters/kubernetes/component/adapter.go:340 +0xda9
github.com/openshift/odo/pkg/devfile/adapters/kubernetes/component.Adapter.Push(0x2ab0fe0, 0xc0008efe60, 0x2a69ce0, 0xc00080bf90, 0xc0002beb40, 0xc00077cbf9, 0x3, 0xc000441560, 0xc0007f4108, 0x7, ...)
	/Users/rajiv/go/src/github.com/openshift/odo/pkg/devfile/adapters/kubernetes/component/adapter.go:153 +0x560
github.com/openshift/odo/pkg/devfile/adapters/kubernetes.Adapter.Push(0x2a1c5c0, 0xc00096c000, 0xc00058b3a0, 0x18, 0x3a7e000, 0x0, 0x0, 0x100, 0x3a7e000, 0x0, ...)
	/Users/rajiv/go/src/github.com/openshift/odo/pkg/devfile/adapters/kubernetes/adapter.go:32 +0xfd
github.com/openshift/odo/pkg/odo/cli/component.(*PushOptions).DevfilePush(0xc00080aa00, 0x0, 0x0)
	/Users/rajiv/go/src/github.com/openshift/odo/pkg/odo/cli/component/devfile.go:66 +0x436
github.com/openshift/odo/pkg/odo/cli/component.(*PushOptions).Run(0xc00080aa00, 0x0, 0x0)
	/Users/rajiv/go/src/github.com/openshift/odo/pkg/odo/cli/component/push.go:130 +0x40
github.com/openshift/odo/pkg/odo/genericclioptions.GenericRun(0x2a4f7e0, 0xc00080aa00, 0xc0006d4900, 0xc000112fc0, 0x0, 0x7)
	/Users/rajiv/go/src/github.com/openshift/odo/pkg/odo/genericclioptions/runnable.go:31 +0x13c
github.com/openshift/odo/pkg/odo/cli/component.NewCmdPush.func1(0xc0006d4900, 0xc000112fc0, 0x0, 0x7)
	/Users/rajiv/go/src/github.com/openshift/odo/pkg/odo/cli/component/push.go:149 +0x5e
github.com/openshift/odo/vendor/github.com/spf13/cobra.(*Command).execute(0xc0006d4900, 0xc000112f50, 0x7, 0x7, 0xc0006d4900, 0xc000112f50)
	/Users/rajiv/go/src/github.com/openshift/odo/vendor/github.com/spf13/cobra/command.go:702 +0x285
github.com/openshift/odo/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc000717b00, 0x286f018, 0xc00083c060, 0x0)
	/Users/rajiv/go/src/github.com/openshift/odo/vendor/github.com/spf13/cobra/command.go:783 +0x2c9
github.com/openshift/odo/vendor/github.com/spf13/cobra.(*Command).Execute(...)
	/Users/rajiv/go/src/github.com/openshift/odo/vendor/github.com/spf13/cobra/command.go:736
main.main()
	/Users/rajiv/go/src/github.com/openshift/odo/cmd/odo/odo.go:75 +0x460
NAME                       READY   STATUS    RESTARTS   AGE
example-7d95d95f5c-rsphf   1/1     Running   0          46s

quarkus

Maven failure because quarkus-maven-plugin:1.3.0 expects maven version us 3.6.2 but the image is using 3.6.1

ODO_COMMAND_RUN is cd ${CHE_PROJECTS_ROOT}/quarkus-quickstarts/getting-started && mvn compile quarkus:dev
+ '[' -z 'cd ${CHE_PROJECTS_ROOT}/quarkus-quickstarts/getting-started && mvn compile quarkus:dev' ']'
+ echo 'ODO_COMMAND_RUN is cd ${CHE_PROJECTS_ROOT}/quarkus-quickstarts/getting-started && mvn compile quarkus:dev'
+ '[' '!' -z ']'
+ sh -c 'cd ${CHE_PROJECTS_ROOT}/quarkus-quickstarts/getting-started && mvn compile quarkus:dev'
---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  23:08 min
[INFO] Finished at: 2020-03-22T23:30:30Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.quarkus:quarkus-maven-plugin:1.3.0.Final:dev (default-cli) on project getting-started: Detected Maven Version (3.6.1)  is not supported, it must be in [3.6.2,). -> [Help 1]

java-mysql

 ◑  Executing run webapp command SPRING_DATASOURCE_URL=jdbc:mysql://db/petclinic \
SPRING_DATASOURCE_USERNAME=petclinic \
SPRING_DATASOURCE_PASSWORD=petclinic \
java -jar -Dspring.profiles.active=mysql \
-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 \
target/*.jar
 ✗  
Unable to exec devrun 

 ✗  Executing run webapp command SPRING_DATASOURCE_URL=jdbc:mysql://db/petclinic \
SPRING_DATASOURCE_USERNAME=petclinic \
SPRING_DATASOURCE_PASSWORD=petclinic \
java -jar -Dspring.profiles.active=mysql \
-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 \
target/*.jar
 [717ms]
 ✗  Failed to start component with name spring-petclinic.
Error: Failed to create the component: error while streaming command: Unauthorized