Script results aren't saved if artifacts aren't found #1472

w3irdrobot · 2019-07-11T20:45:13Z

Is this a BUG REPORT or FEATURE REQUEST?:

Bug.

What happened:

When running a step to install node dependencies, the install failed, which is fine. However, the stdout from that step telling me that the package.json wasn't present wasn't put into the outputs.result in the NodeStatus struct like I'd expect.

What you expected to happen:

I expected outputs.result to have the logs from the container in it.

How to reproduce it (as minimally and precisely as possible):

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: node-install-fail
spec:
  entrypoint: entrypoint
  templates:
  - name: entrypoint
    steps:
    - - name: install
        template: install-deps

  - name: install-deps
    script:
      image: node:9.1-alpine
      command: [sh]
      source: |
        npm install
    outputs:
      artifacts:
      - name: dependencies
        path: node_modules

Run this and then look at the Node Status of the install-deps step. It has a message saying it can't save the output but there isn't any result showing the logs npm install outputted.

Environment:

Argo version:

$ argo version
argo: v2.3.0
  BuildDate: 2019-05-20T22:11:23Z
  GitCommit: 88fcc70dcf6e60697e6716edc7464a403c49b27e
  GitTreeState: clean
  GitTag: v2.3.0
  GoVersion: go1.11.5
  Compiler: gc
  Platform: darwin/amd64

Kubernetes version :

$ kubectl version -o yaml
clientVersion:
  buildDate: "2019-06-20T04:49:16Z"
  compiler: gc
  gitCommit: e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529
  gitTreeState: clean
  gitVersion: v1.15.0
  goVersion: go1.12.6
  major: "1"
  minor: "15"
  platform: darwin/amd64
serverVersion:
  buildDate: "2019-06-19T16:32:14Z"
  compiler: gc
  gitCommit: e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529
  gitTreeState: clean
  gitVersion: v1.15.0
  goVersion: go1.12.5
  major: "1"
  minor: "15"
  platform: linux/amd64

Other debugging information (if applicable):

workflow result:

$ argo get node-install-fail
Name:                node-install-fail
Namespace:           default
ServiceAccount:      default
Status:              Failed
Message:             child 'node-install-fail-2024117197' failed
Created:             Thu Jul 11 16:34:35 -0400 (4 minutes ago)
Started:             Thu Jul 11 16:34:35 -0400 (4 minutes ago)
Finished:            Thu Jul 11 16:34:38 -0400 (4 minutes ago)
Duration:            3 seconds

STEP                  PODNAME                       DURATION  MESSAGE
 ✖ node-install-fail                                          child 'node-install-fail-2024117197' failed
 └---⚠ install        node-install-fail-2024117197  2s        failed to save outputs: path node_modules does not exist (or node_modules is empty) in archive /argo/outputs/artifacts/dependencies.tgz

executor logs:

$ kubectl logs node-install-fail-2024117197 -c init
time="2019-07-11T20:34:36Z" level=info msg="Creating a docker executor"
time="2019-07-11T20:34:36Z" level=info msg="Executor (version: v2.3.0-rc1, build_date: 2019-04-10T15:20:56Z) initialized with template:\n{\"name\":\"install-deps\",\"inputs\":{},\"outputs\":{\"artifacts\":[{\"name\":\"dependencies\",\"path\":\"node_modules\"}]},\"metadata\":{},\"script\":{\"name\":\"\",\"image\":\"node:9.1-alpine\",\"command\":[\"sh\"],\"resources\":{},\"source\":\"npm install\\n\"},\"archiveLocation\":{\"s3\":{\"endpoint\":\"minio.platform.svc.cluster.local:9000\",\"bucket\":\"argo-artifacts\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"minio\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"minio\",\"key\":\"secretkey\"},\"key\":\"node-install-fail/node-install-fail-2024117197\"}}}"
time="2019-07-11T20:34:36Z" level=info msg="Loading script source to /argo/staging/script"
time="2019-07-11T20:34:36Z" level=info msg="Start loading input artifacts..."
time="2019-07-11T20:34:36Z" level=info msg="Alloc=3623 TotalAlloc=9215 Sys=70078 NumGC=3 Goroutines=3"

$ kubectl logs node-install-fail-2024117197 -c init
time="2019-07-11T20:34:36Z" level=info msg="Creating a docker executor"
time="2019-07-11T20:34:36Z" level=info msg="Executor (version: v2.3.0-rc1, build_date: 2019-04-10T15:20:56Z) initialized with template:\n{\"name\":\"install-deps\",\"inputs\":{},\"outputs\":{\"artifacts\":[{\"name\":\"dependencies\",\"path\":\"node_modules\"}]},\"metadata\":{},\"script\":{\"name\":\"\",\"image\":\"node:9.1-alpine\",\"command\":[\"sh\"],\"resources\":{},\"source\":\"npm install\\n\"},\"archiveLocation\":{\"s3\":{\"endpoint\":\"minio.platform.svc.cluster.local:9000\",\"bucket\":\"argo-artifacts\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"minio\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"minio\",\"key\":\"secretkey\"},\"key\":\"node-install-fail/node-install-fail-2024117197\"}}}"
time="2019-07-11T20:34:36Z" level=info msg="Loading script source to /argo/staging/script"
time="2019-07-11T20:34:36Z" level=info msg="Start loading input artifacts..."
time="2019-07-11T20:34:36Z" level=info msg="Alloc=3623 TotalAlloc=9215 Sys=70078 NumGC=3 Goroutines=3"

workflow-controller logs:

$ kubectl logs -n kube-system $(kubectl get pods -l app=workflow-controller -n kube-system -o name)
time="2019-07-11T20:43:11Z" level=info msg="Processing workflow" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:11Z" level=info msg="Updated phase  -> Running" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:11Z" level=info msg="Steps node node-install-fail (node-install-fail) initialized Running" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:11Z" level=info msg="StepGroup node node-install-fail[0] (node-install-fail-816242364) initialized Running" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:11Z" level=info msg="Created pod: node-install-fail[0].install (node-install-fail-2024117197)" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:11Z" level=info msg="Pod node node-install-fail[0].install (node-install-fail-2024117197) initialized Pending" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:11Z" level=info msg="Workflow step group node node-install-fail[0] (node-install-fail-816242364) not yet completed" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:11Z" level=info msg="Workflow update successful" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:12Z" level=info msg="Processing workflow" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:12Z" level=info msg="Updating node node-install-fail[0].install (node-install-fail-2024117197) message: PodInitializing"
time="2019-07-11T20:43:12Z" level=info msg="Workflow step group node node-install-fail[0] (node-install-fail-816242364) not yet completed" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:12Z" level=info msg="Workflow update successful" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:13Z" level=info msg="Processing workflow" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:13Z" level=info msg="Workflow step group node node-install-fail[0] (node-install-fail-816242364) not yet completed" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:14Z" level=info msg="Processing workflow" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:14Z" level=info msg="Workflow step group node node-install-fail[0] (node-install-fail-816242364) not yet completed" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:15Z" level=info msg="Processing workflow" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:15Z" level=info msg="Updating node node-install-fail[0].install (node-install-fail-2024117197) status Pending -> Running"
time="2019-07-11T20:43:15Z" level=info msg="Workflow step group node node-install-fail[0] (node-install-fail-816242364) not yet completed" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:15Z" level=info msg="Workflow update successful" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:16Z" level=info msg="Processing workflow" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:16Z" level=info msg="Workflow step group node node-install-fail[0] (node-install-fail-816242364) not yet completed" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:16Z" level=info msg="Processing workflow" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:16Z" level=info msg="Updating node node-install-fail[0].install (node-install-fail-2024117197) status Running -> Error"
time="2019-07-11T20:43:16Z" level=info msg="Updating node node-install-fail[0].install (node-install-fail-2024117197) message: failed to save outputs: path node_modules does not exist (or node_modules is empty) in archive /argo/outputs/artifacts/dependencies.tgz"
time="2019-07-11T20:43:16Z" level=info msg="Step group node node-install-fail[0] (node-install-fail-816242364) deemed failed: child 'node-install-fail-2024117197' failed" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:16Z" level=info msg="node node-install-fail[0] (node-install-fail-816242364) phase Running -> Failed" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:16Z" level=info msg="node node-install-fail[0] (node-install-fail-816242364) message: child 'node-install-fail-2024117197' failed" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:16Z" level=info msg="node node-install-fail[0] (node-install-fail-816242364) finished: 2019-07-11 20:43:16.341420672 +0000 UTC" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:16Z" level=info msg="step group node-install-fail-816242364 was unsuccessful: child 'node-install-fail-2024117197' failed" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:16Z" level=info msg="Outbound nodes of node-install-fail-2024117197 is [node-install-fail-2024117197]" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:16Z" level=info msg="Outbound nodes of node-install-fail is [node-install-fail-2024117197]" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:16Z" level=info msg="node node-install-fail (node-install-fail) phase Running -> Failed" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:16Z" level=info msg="node node-install-fail (node-install-fail) message: child 'node-install-fail-2024117197' failed" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:16Z" level=info msg="node node-install-fail (node-install-fail) finished: 2019-07-11 20:43:16.341458093 +0000 UTC" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:16Z" level=info msg="Checking daemoned children of node-install-fail" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:16Z" level=info msg="Updated phase Running -> Failed" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:16Z" level=info msg="Updated message  -> child 'node-install-fail-2024117197' failed" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:16Z" level=info msg="Marking workflow completed" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:16Z" level=info msg="Checking daemoned children of " namespace=default workflow=node-install-fail
time="2019-07-11T20:43:16Z" level=info msg="Workflow update successful" namespace=default workflow=node-install-fail
time="2019-07-11T20:43:17Z" level=info msg="Labeled pod default/node-install-fail-2024117197 completed"

I believe I know what the issue is and will be submitting a PR soon.

The text was updated successfully, but these errors were encountered:

sarabala1979 · 2019-07-12T21:26:22Z

it is workflow definition issue. Code change doesn't make any impact. Try below changes

To upload artifacts:

kind: Workflow
metadata:
  name: node-install-fail
spec:
  entrypoint: entrypoint
  templates:
  - name: entrypoint
    steps:
    - - name: install
        template: install-deps

  - name: install-deps
    script:
      image: node:9.1-alpine
      command: [sh]
      source: |
        npm install > /tmp/node_modules.txt
    outputs:
      artifacts:
      - name: dependencies
        path: /tmp/node_modules.txt

if you want outputs.result

 apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: node-install-fail
spec:
  entrypoint: entrypoint
  templates:
  - name: entrypoint
    steps:
    - - name: install
        template: install-deps

  - name: install-deps
    script:
      image: node:9.1-alpine
      command: [sh]
      source: |
        npm install

I can see the output

        result: |-
          + test@0.6.0
          added 2 packages in 1.043s
      phase: Succeeded

w3irdrobot · 2019-07-12T22:19:27Z

Yep. The issue is the fact the path for the output doesn't exist. However, it would be very beneficial to have the output.result even if the artifacts don't exist as a way to diagnose the problem. To make this happen, a code change would be required.

jessesuen · 2019-07-17T21:19:14Z

@searsaw we are actually changing the behavior of conditionally storing the script's outputs, depending on if the script output is actually used some place else. The reason for this is: #1359.

Even if #1474 is merged, it would not solve your problem since with the new behavior, output.result would not be stored in the workflow as it is not referenced any place else in the workflow.

That said, if the script's output.result is referenced later in the workflow, then PR #1474 would help.

alexec · 2020-05-12T22:57:22Z

Is this still a problem please?

w3irdrobot · 2020-05-12T23:12:26Z

@alexec I think this is a nonissue at this point.

alexec · 2020-05-13T01:27:12Z

Ok. Closing.

…ixes argoproj#1472

…ixes #1472 (#3025)

w3irdrobot mentioned this issue Jul 12, 2019

Moved executor script capture to before artifact collection #1474

Closed

sarabala1979 added type/bug and removed type/bug labels Jul 17, 2019

sarabala1979 added type/bug question labels Jul 30, 2019

alexec added the artifacts label May 12, 2020

alexec closed this as completed May 13, 2020

alexec added a commit to alexec/argo-workflows that referenced this issue May 13, 2020

fix(executor): Save script results before artifacts in case of error. F…

6214fd6

…ixes argoproj#1472

alexec mentioned this issue May 13, 2020

fix(executor): Save script results before artifacts in case of error. Fixes #1472 #3025

Merged

6 tasks

alexec added a commit that referenced this issue May 14, 2020

fix(executor): Save script results before artifacts in case of error. F…

f17e946

…ixes #1472 (#3025)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Script results aren't saved if artifacts aren't found #1472

Script results aren't saved if artifacts aren't found #1472

w3irdrobot commented Jul 11, 2019

sarabala1979 commented Jul 12, 2019 •

edited

w3irdrobot commented Jul 12, 2019

jessesuen commented Jul 17, 2019

alexec commented May 12, 2020

w3irdrobot commented May 12, 2020

alexec commented May 13, 2020

Script results aren't saved if artifacts aren't found #1472

Script results aren't saved if artifacts aren't found #1472

Comments

w3irdrobot commented Jul 11, 2019

sarabala1979 commented Jul 12, 2019 • edited

w3irdrobot commented Jul 12, 2019

jessesuen commented Jul 17, 2019

alexec commented May 12, 2020

w3irdrobot commented May 12, 2020

alexec commented May 13, 2020

sarabala1979 commented Jul 12, 2019 •

edited