sd-local - ability to run steps inside build container

[jithine]

What happened:

Currently sd-local allows developers to run an entire job in build container. It will be beneficial to development workflow to get into the environment of sd-local so that they can run steps and commands inside the container and debug any failures.

What you expected to happen:

1. sd-local to support an exec mode into the build container where they can run any steps they want.
2. Similar to docker run -it container sh
3. Developers gets an interactive shell inside the environment setup by sd-local

How to reproduce it:

N/A

Reference

• Local build feature #1314

[jithine]

cc @ibu1224 @kkisic @MysticDoll @kumada626

[ibu1224]

@jithine
Hello.
Let me ask you a question for the this feature.

In order to debug, the user execute an arbitrary command or script.

We need to think when that command will be executed.
In the case of sd-local, the following cases are possible.

Cases

1. Before executing steps of screwdriver.yaml
2. After executing all steps of screwdriver.yaml

About 1.
In this case, user command executing before executing the steps of screwdriver.yaml.
Maybe we can check the operation in the environment before the steps is executed.

About 2.
In this case, the user's command is executed after steps of screwdriver.yaml is executed.
Maybe able to see the results after the steps of screwdriver.yaml are executed

Which of the above cases would you need to solve this issue? Or both? Or not?

Tell me what you think 🙏

[yuichi10]

I want some function that makes it easy to run step commands.
For example, all step commands are preserved as alias and user can exec a step command by using the alias.

If there is a YAML as shown below

jobs:
  main:
    requires: [ ~commit ]
    image: node10:latest
    steps:
      - test: npm test
      - echo: echo 'success'

user can exec steps as shown below

$ sd-echo
success
$ sd-test 
...

[jithine]

@ibu1224 intention is to keep it separate from running of steps, which will be a separate feature than this.

I echo what @yuichi10 has mentioned. Ability to to run individual build step once inside the container, and also run any command once inside the container.

[ibu1224]

@jithine Thank you for the reply.
Please let me confirm the following.
The build container in which the user executes any command should be after setup(sd-setup-launcher), right?

The implementation direction seems to be (About 1).
Allows any commands to be executed in the build container.

@jithine , @screwdriver-cd/yahoo-japan
If you have any questions, please comment.
The following is a summary of the specifications　and implementation order.

STEP1 Ability to launch a build container and enter interactive mode.

Create a new command run.　Similar to the docker run.
Run the following command to enter the build container.

$ sd-local run <jobName>

The image of the build container uses the image of the job.
The build container is after executing sd-setup-launcher.
If possible, remove the build container when the user leaves the build container.

The same flags as in the build command should be supported.
For example, --sudo flag.

STEP2 In the build container, create alias to step of screwdriver.yaml.

Related to #2238 (comment)
In the build container, to execute the steps of the screwdriver.yaml
Create an alias for each step.

If there is a screwdriver.yaml as shown below

jobs:
  main:
    requires: [ ~commit ]
    image: node10:latest
    steps:
      - test: echo 'test'
      - echo 'success'

user can exec steps in build container as shown below

$ sd-test
=> test
$ sd-step-2
=> success

[yoshwata]

Does sd-local run up with the state that bookends for setup is done?

[ibu1224]

@yoshwata
Review thank you.
The setup for the sd-local run command is the sd-launcher only.
The same is true for the sd-local build command.

Maybe there is no bookends feature in sd-local yet.

[yoshwata]

@ibu1224 I understand that sd-local run does sd-setup-launcher bookend like sd-local build does.

[ibu1224]

@yoshwata Yes. The build container is after executing sd-setup-launcher.
I added it to the #2238 (comment) above.

[tk3fftk]

About cmd name, it seems run and build look similar and confusable for users especially new ones. How about sd-local interact or sd-local build -i (--interactive)? (I originally mentioned in screwdriver-cd/sd-local#45 (comment) )

[wahapo]

I think 2. $ sd-local build -i <jobName> seems to me to be the most rational. Considering that the build step can be executed interactively, it's not too strange to have it as an option in the build command. In addition to that, the other optional parameters are the same and therefore easier for the user to understand.

1. $ sd-local run <jobName>
2. $ sd-local build -i <jobName>
3. $ sd-local interact <jobName>

I'm thinking of changing the specification to $ sd-local build -i <jobName>.

[jithine]

I like the interactive option for build command, option 2.

[jithine]

@wahapo when I tried locally, it didn't quite enter interactive mode properly.

❯ sd-local build -i doc_build
INFO   [0000] Prepare to start build...
INFO   [0007] Pulling docker image from ...
/opt/sd/local_run.sh "{\"id\":0,\"environme ...........
Welcome to sd-local interactive mode. If you exit type 'exit'

>
> 'exit'
>
> ls
> ls
> sh: unexpected EOF while looking for matching `"'
sh: syntax error: unexpected end of file
sh-4.2# ls
bin  boot  dev	etc  home  lib	lib64  media  mnt  opt	proc  root  run  sbin  sd  srv	sys  tmp  usr  var

It felt like local_run.sh was waiting for a string input to complete or something.

Also the message should be fixed to If you exit type 'exit' => To exit type 'exit'

[wahapo]

It looks like the command string is broken in the middle when the json string is long. I'll try to fix it.

[ibu1224]

@jithine
I fixed it with the following PR
screwdriver-cd/sd-local#49

Please comment if you have any questions.

[jithine]

@ibu1224 I tested and it works 👍