Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PNS executor (2.9.4) having issue on windows with input/output artifacts #4009

Closed
guoweis-work opened this issue Sep 12, 2020 · 7 comments
Closed
Labels

Comments

@guoweis-work
Copy link

guoweis-work commented Sep 12, 2020

Summary

I'm using a cross-platform workflow. It seems working as long as I don't have artifacts. Once I do, it gives me error like failed to save outputs: verify serviceaccount rolling:workflow-runner has necessary privileges

The same serviceaccount works fine for linux box and I'm not sure this is really a privilege problem.

Diagnostics

argo 2.9.4

wait container output

time="2020-09-12T01:41:37.965Z" level=info msg="Creating PNS executor (namespace: rolling, pod: full-loop-test-d5qrt-1757493819, pid: 15484, hasOutputs: true)"
time="2020-09-12T01:41:37.965Z" level=info msg="Executor (version: v2.9.4+20d2ace.dirty, build_date: 2020-07-24T21:51:02Z) initialized (pod: rolling/full-loop-test-d5qrt-1757493819) with template:\n{\"name\":\"class-lm\",\"arguments\":{},\"inputs\":{\"parameters\":[{\"name\":\"orgID\",\"value\":\"asrpipelinetest-rolling\"}],\"artifacts\":[{\"name\":\"workDir\",\"path\":\"/data\",\"s3\":{\"endpoint\":\"minio:9000\",\"bucket\":\"my-bucket\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"minio-creds\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"minio-creds\",\"key\":\"secretkey\"},\"key\":\"full-loop-test-d5qrt/full-loop-test-d5qrt-669130731/modifiedWorkDir.tgz\"}}]},\"outputs\":{\"artifacts\":[{\"name\":\"modifiedWorkDir\",\"path\":\"/data\"}]},\"nodeSelector\":{\"beta.kubernetes.io/os\":\"windows\"},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"outreachmeeting.azurecr.io/asrtool-classlm:20200727.14-0939787e42-master\",\"command\":[\"c:\\\\ConvertToCustomData\\\\ConvertToCustomData.exe\"],\"args\":[\"--input\",\"c:\\\\data\\\\asrpipelinetest-rolling\\\\classlm\\\\inputDgiData.json\",\"--output\",\"c:\\\\data\\\\asrpipelinetest-rolling\\\\classlm\\\\classlm.txt\",\"--inputtype\",\"dgi\"],\"resources\":{},\"imagePullPolicy\":\"IfNotPresent\"},\"archiveLocation\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio:9000\",\"bucket\":\"my-bucket\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"minio-creds\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"minio-creds\",\"key\":\"secretkey\"},\"key\":\"full-loop-test-d5qrt/full-loop-test-d5qrt-1757493819\"}}}"
time="2020-09-12T01:41:37.965Z" level=warning msg="Polling root processes (1m0s)"
time="2020-09-12T01:41:37.966Z" level=error msg="executor error: open /: Access is denied.\ngithub.com/argoproj/argo/errors.Wrap\n\tC:/Users/ContainerAdministrator/go/src/github.com/argoproj/argo/errors/errors.go:88\ngithub.com/argoproj/argo/errors.InternalWrapError\n\tC:/Users/ContainerAdministrator/go/src/github.com/argoproj/argo/errors/errors.go:71\ngithub.com/argoproj/argo/workflow/executor/pns.(*PNSExecutor).WaitInit\n\tC:/Users/ContainerAdministrator/go/src/github.com/argoproj/argo/workflow/executor/pns/pns.go:156\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).Wait\n\tC:/Users/ContainerAdministrator/go/src/github.com/argoproj/argo/workflow/executor/executor.go:896\ngithub.com/argoproj/argo/cmd/argoexec/commands.waitContainer\n\tC:/Users/ContainerAdministrator/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:40\ngithub.com/argoproj/argo/cmd/argoexec/commands.NewWaitCommand.func1\n\tC:/Users/ContainerAdministrator/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:16\ngithub.com/spf13/cobra.(*Command).execute\n\tC:/Users/ContainerAdministrator/go/pkg/mod/github.com/spf13/cobra@v0.0.4-0.20181021141114-fe5e611709b0/command.go:766\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tC:/Users/ContainerAdministrator/go/pkg/mod/github.com/spf13/cobra@v0.0.4-0.20181021141114-fe5e611709b0/command.go:852\ngithub.com/spf13/cobra.(*Command).Execute\n\tC:/Users/ContainerAdministrator/go/pkg/mod/github.com/spf13/cobra@v0.0.4-0.20181021141114-fe5e611709b0/command.go:800\nmain.main\n\tC:/Users/ContainerAdministrator/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:17\nruntime.main\n\tc:/go/src/runtime/proc.go:203\nruntime.goexit\n\tc:/go/src/runtime/asm_amd64.s:1357"
time="2020-09-12T01:41:37.966Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2020-09-12T01:41:37.966Z" level=info msg="Capturing script exit code"
time="2020-09-12T01:41:37.967Z" level=warning msg="Failed to stat /proc/0/root: CreateFile /proc/0/root: The system cannot find the path specified."
time="2020-09-12T01:41:37.967Z" level=warning msg="Failed to stat /proc/4/root: CreateFile /proc/4/root: The system cannot find the path specified."
time="2020-09-12T01:41:38.003Z" level=info msg="Getting exit code of "
time="2020-09-12T01:41:38.007Z" level=info msg="Saving logs"
time="2020-09-12T01:41:38.020Z" level=warning msg="Failed to stat /proc/0/root: CreateFile /proc/0/root: The system cannot find the path specified."
time="2020-09-12T01:41:38.020Z" level=warning msg="Failed to stat /proc/4/root: CreateFile /proc/4/root: The system cannot find the path specified."
time="2020-09-12T01:41:38.072Z" level=warning msg="Failed to stat /proc/0/root: CreateFile /proc/0/root: The system cannot find the path specified."
time="2020-09-12T01:41:38.072Z" level=warning msg="Failed to stat /proc/4/root: CreateFile /proc/4/root: The system cannot find the path specified."
time="2020-09-12T01:41:38.127Z" level=warning msg="Failed to stat /proc/0/root: CreateFile /proc/0/root: The system cannot find the path specified."
time="2020-09-12T01:41:38.127Z" level=warning msg="Failed to stat /proc/4/root: CreateFile /proc/4/root: The system cannot find the path specified."
time="2020-09-12T01:41:38.138Z" level=error msg="executor error: container \"main\" in pod \"full-loop-test-d5qrt-1757493819\" is waiting to start: PodInitializing"
time="2020-09-12T01:41:38.138Z" level=info msg="Killing sidecars"
time="2020-09-12T01:41:38.142Z" level=info msg="Alloc=5690 TotalAlloc=15796 Sys=16394 NumGC=5 Goroutines=7"
panic: open /dev/termination-log: The system cannot find the path specified.

goroutine 1 [running]:
github.com/argoproj/argo/util.WriteTeriminateMessage(...)
	C:/Users/ContainerAdministrator/go/src/github.com/argoproj/argo/util/util.go:63
github.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).HandleError(0xc0003ce600)
	C:/Users/ContainerAdministrator/go/src/github.com/argoproj/argo/workflow/executor/executor.go:117 +0x433
github.com/argoproj/argo/cmd/argoexec/commands.waitContainer(0x202c9e0, 0xc0002d5ea0)
	C:/Users/ContainerAdministrator/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:62 +0x67e
github.com/argoproj/argo/cmd/argoexec/commands.NewWaitCommand.func1(0xc0002e2280, 0x2ec3970, 0x0, 0x0)
	C:/Users/ContainerAdministrator/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:16 +0x2d
github.com/spf13/cobra.(*Command).execute(0xc0002e2280, 0x2ec3970, 0x0, 0x0, 0xc0002e2280, 0x2ec3970)
	C:/Users/ContainerAdministrator/go/pkg/mod/github.com/spf13/cobra@v0.0.4-0.20181021141114-fe5e611709b0/command.go:766 +0x2b1
github.com/spf13/cobra.(*Command).ExecuteC(0xc0003f3900, 0xc00009ff50, 0xc0008d7f50, 0x405aa6)
	C:/Users/ContainerAdministrator/go/pkg/mod/github.com/spf13/cobra@v0.0.4-0.20181021141114-fe5e611709b0/command.go:852 +0x2f1
github.com/spf13/cobra.(*Command).Execute(...)
	C:/Users/ContainerAdministrator/go/pkg/mod/github.com/spf13/cobra@v0.0.4-0.20181021141114-fe5e611709b0/command.go:800
main.main()
	C:/Users/ContainerAdministrator/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:17 +0x32
  - name: class-lm
    nodeSelector:
      "beta.kubernetes.io/os": windows
    inputs:
      parameters:
      - name: orgID
      artifacts:
      - name: workDir
        path: /data
    container:
      image: somecontainer:20200727.14-0939787e42-master
      imagePullPolicy: IfNotPresent
      command: ["c:\\some\\thing.exe"]
      args:
      - "--input"
      - "\\data\\{{inputs.parameters.orgID}}\\mydir\\input.json"
      - "--output"
      - "\\data\\{{inputs.parameters.orgID}}\\mydir\\myoutput.txt"
      - "--inputtype"
      - "dgi"
    outputs:
      artifacts:
      - name: modifiedWorkDir
        path: /data

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

@guoweis-work
Copy link
Author

guoweis-work commented Sep 12, 2020

nvm. this seems causing by that i use the same folder for both input/output artifacts. This works on linux, but not windows.

@guoweis-work
Copy link
Author

reopen this, I still like this to be looked at. I start wondering if output artifact is even supported on windows platform.

@alexec
Copy link
Contributor

alexec commented Sep 14, 2020

Windows work: Fixes #1507 and #1383 (#2747)

@alexec
Copy link
Contributor

alexec commented Sep 14, 2020

@lippertmarkus could you please help?

@lippertmarkus
Copy link
Member

PNS doesn't work with Windows containers, see https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#v1-pod

V1.Pod.shareProcessNamespace - this is a beta feature, and depends on Linux namespaces which are not implemented on Windows. Windows cannot share process namespaces or the container's root filesystem. Only the network can be shared.

Input/Output artifacts should work on Windows. I just tried the following workflow with the docker executor and Argo v2.10.1 without a problem. Using the same folder for input/output artifact also works:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artifact-passing-
  namespace: default
spec:
  entrypoint: artifact-example
  templates:
  - name: artifact-example
    steps:
    - - name: generate-artifact
        template: whalesay
    - - name: consume-artifact
        template: print-message
        arguments:
          artifacts:
          - name: message
            from: "{{steps.generate-artifact.outputs.artifacts.hello-art}}"
    - - name: consume-win-artifact
        template: consumewinoutput
        arguments:
          artifacts:
          - name: message2
            from: "{{steps.consume-artifact.outputs.artifacts.modifiedWorkDir}}"

  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [sh, -c]
      args: ["cowsay hello world | tee /tmp/hello_world.txt"]
    nodeSelector:
      kubernetes.io/os: linux
    outputs:
      artifacts:
      - name: hello-art
        path: /tmp/

  - name: print-message
    inputs:
      artifacts:
      - name: message
        path: /tmp/
    nodeSelector:
      kubernetes.io/os: windows
    container:
      image: mcr.microsoft.com/windows/servercore:1809
      command: ["cmd", "/c"]
      args: ["dir c:\\tmp"]
    outputs:
      artifacts:
      - name: modifiedWorkDir
        path: /tmp/

  - name: consumewinoutput
    container:
      image: docker/whalesay:latest
      command: [sh, -c]
      args: ["ls /tmp/message2"]
    nodeSelector:
      kubernetes.io/os: linux
    inputs:
      artifacts:
      - name: message2
        path: /tmp/message2

@lippertmarkus
Copy link
Member

@guoweis-outreach I added information about this limitation to the docs. Is there anything else or can we close this issue?

@stale
Copy link

stale bot commented Dec 8, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Dec 8, 2020
@stale stale bot closed this as completed Dec 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants