Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Don't return an error when getting a nonexistent k8s resource #1823

Closed
dmayle opened this issue Dec 8, 2019 · 3 comments
Closed

Don't return an error when getting a nonexistent k8s resource #1823

dmayle opened this issue Dec 8, 2019 · 3 comments
Assignees
@dmayle
Labels
type/feature Feature request

Comments

@dmayle
Copy link
Contributor

dmayle commented Dec 8, 2019

Is this a BUG REPORT or FEATURE REQUEST?:
This is somewhere in between the two.

What happened:
When you try to check for the existence of a resource, the resource executor passes through the error directly from kubectl. I've filed an issue against kubernetes ( kubernetes/kubernetes#86042 ) because the get command returns an error when querying the singular if a resource doesn't exist.

What you expected to happen:
I should be able to get a response without the workflow step failing

How to reproduce it (as minimally and precisely as possible):

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: minimal-test-case
  namespace: argo-workflows
spec:
  entrypoint: workflow
  templates:
  - name: workflow
    steps:
    - - name: check-for-existence
        template: check-for-existence
  - name: check-for-existence
    resource:
      action: get
      manifest: |
        apiVersion: v1
        kind: Pod
        metadata:
          name: does-not-exist
          namespace: argo-workflows
    outputs:
      parameters:
      - name: not-found
        valueFrom:
          jsonPath: '{.metadata.name}'

Anything else we need to know?:
I'm happy to try fixing it myself, just want to make sure we agree on the approach

Environment:

  • Argo version:
$ argo version
argo: v2.4.2
  BuildDate: 2019-10-21T18:39:29Z
  GitCommit: 675c66267f0c916de0f233d8101aa0646acb46d4
  GitTreeState: clean
  GitTag: v2.4.2
  GoVersion: go1.11.5
  Compiler: gc
  Platform: linux/amd64
  • Kubernetes version :
$ kubectl version -o yaml
clientVersion:
  buildDate: "2019-10-15T19:18:23Z"
  compiler: gc
  gitCommit: c97fe5036ef3df2967d086711e6c0c405941e14b
  gitTreeState: clean
  gitVersion: v1.16.2
  goVersion: go1.12.10
  major: "1"
  minor: "16"
  platform: linux/amd64
serverVersion:
  buildDate: "2019-08-19T11:05:50Z"
  compiler: gc
  gitCommit: 2d3c76f9091b6bec110a5e63777c332469e0cba2
  gitTreeState: clean
  gitVersion: v1.15.3
  goVersion: go1.12.9
  major: "1"
  minor: "15"
  platform: linux/amd64

Other debugging information (if applicable):

  • workflow result:
$ argo get <workflowname>
Name:                minimal-test-caseb78sd
Namespace:           argo-workflows
ServiceAccount:      argo
Status:              Failed
Message:             child 'minimal-test-caseb78sd-3318963475' failed
Created:             Sun Dec 08 20:27:50 +0100 (4 seconds ago)
Started:             Sun Dec 08 20:27:50 +0100 (4 seconds ago)
Finished:            Sun Dec 08 20:27:54 +0100 (now)
Duration:            4 seconds

STEP                                              PODNAME                            DURATION  MESSAGE
 ✖ minimal-test-caseb78sd (workflow)                                                           child 'minimal-test-caseb78sd-3318963475' failed
 └---✖ check-for-existence (check-for-existence)  minimal-test-caseb78sd-3318963475  4s        Error from server (NotFound): pods "some-pv" not found
  • executor logs:
$ kubectl logs <failedpodname> -c init
$ kubectl logs <failedpodname> -c wait
  • workflow-controller logs:
$ kubectl logs -n argo $(kubectl get pods -l app=workflow-controller -n argo -o name)
time="2019-12-08T19:27:50Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=minimal-test-caseb78sd                                                 
time="2019-12-08T19:27:50Z" level=info msg="Updated phase  -> Running" namespace=argo-workflows workflow=minimal-test-caseb78sd                                                                           
time="2019-12-08T19:27:50Z" level=info msg="Steps node minimal-test-caseb78sd (minimal-test-caseb78sd) initialized Pending" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:50Z" level=info msg="Create stored template '/workflow'" namespace=argo-workflows workflow=minimal-test-caseb78sd                                                                                    
time="2019-12-08T19:27:50Z" level=info msg="node minimal-test-caseb78sd (minimal-test-caseb78sd) phase Pending -> Running" namespace=argo-workflows workflow=minimal-test-caseb78sd                           
time="2019-12-08T19:27:50Z" level=info msg="StepGroup node minimal-test-caseb78sd[0] (minimal-test-caseb78sd-3360048478) initialized Running" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:50Z" level=info msg="Pod node minimal-test-caseb78sd[0].check-for-existence (minimal-test-caseb78sd-3318963475) initialized Pending" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:50Z" level=info msg="Create stored template '/check-for-existence'" namespace=argo-workflows workflow=minimal-test-caseb78sd                                                                         
time="2019-12-08T19:27:50Z" level=info msg="Created pod: minimal-test-caseb78sd[0].check-for-existence (minimal-test-caseb78sd-3318963475)" namespace=argo-workflows workflow=minimal-test-caseb78sd          
time="2019-12-08T19:27:50Z" level=info msg="Workflow step group node minimal-test-caseb78sd[0] (minimal-test-caseb78sd-3360048478) not yet completed" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:50Z" level=info msg="Workflow update successful" namespace=argo-workflows workflow=minimal-test-caseb78sd                                          
time="2019-12-08T19:27:51Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=minimal-test-caseb78sd                                                                                                   
time="2019-12-08T19:27:51Z" level=info msg="Updating node minimal-test-caseb78sd[0].check-for-existence (minimal-test-caseb78sd-3318963475) message: ContainerCreating"                                       
time="2019-12-08T19:27:51Z" level=info msg="Skipped pod minimal-test-caseb78sd[0].check-for-existence (minimal-test-caseb78sd-3318963475) creation: already exists" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:51Z" level=info msg="Workflow step group node minimal-test-caseb78sd[0] (minimal-test-caseb78sd-3360048478) not yet completed" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:51Z" level=info msg="Workflow update successful" namespace=argo-workflows workflow=minimal-test-caseb78sd                                                                                            
time="2019-12-08T19:27:52Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=minimal-test-caseb78sd                                                                                     
time="2019-12-08T19:27:52Z" level=info msg="Skipped pod minimal-test-caseb78sd[0].check-for-existence (minimal-test-caseb78sd-3318963475) creation: already exists" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:52Z" level=info msg="Workflow step group node minimal-test-caseb78sd[0] (minimal-test-caseb78sd-3360048478) not yet completed" namespace=argo-workflows workflow=minimal-test-caseb78sd              
time="2019-12-08T19:27:52Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=minimal-test-caseb78sd                                                                                     
time="2019-12-08T19:27:52Z" level=info msg="Updating node minimal-test-caseb78sd[0].check-for-existence (minimal-test-caseb78sd-3318963475) status Pending -> Running"
time="2019-12-08T19:27:52Z" level=info msg="Skipped pod minimal-test-caseb78sd[0].check-for-existence (minimal-test-caseb78sd-3318963475) creation: already exists" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:52Z" level=info msg="Workflow step group node minimal-test-caseb78sd[0] (minimal-test-caseb78sd-3360048478) not yet completed" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:52Z" level=info msg="Workflow update successful" namespace=argo-workflows workflow=minimal-test-caseb78sd                                                                                              
time="2019-12-08T19:27:53Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=minimal-test-caseb78sd                                                            
time="2019-12-08T19:27:53Z" level=info msg="Skipped pod minimal-test-caseb78sd[0].check-for-existence (minimal-test-caseb78sd-3318963475) creation: already exists" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:53Z" level=info msg="Workflow step group node minimal-test-caseb78sd[0] (minimal-test-caseb78sd-3360048478) not yet completed" namespace=argo-workflows workflow=minimal-test-caseb78sd                
time="2019-12-08T19:27:53Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=minimal-test-caseb78sd           
time="2019-12-08T19:27:54Z" level=info msg="Skipped pod minimal-test-caseb78sd[0].check-for-existence (minimal-test-caseb78sd-3318963475) creation: already exists" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:54Z" level=info msg="Workflow step group node minimal-test-caseb78sd[0] (minimal-test-caseb78sd-3360048478) not yet completed" namespace=argo-workflows workflow=minimal-test-caseb78sd      
time="2019-12-08T19:27:54Z" level=info msg="Processing workflow" namespace=argo-workflows workflow=minimal-test-caseb78sd                                                                       
time="2019-12-08T19:27:54Z" level=info msg="Updating node minimal-test-caseb78sd[0].check-for-existence (minimal-test-caseb78sd-3318963475) status Running -> Failed"                    
time="2019-12-08T19:27:54Z" level=info msg="Updating node minimal-test-caseb78sd[0].check-for-existence (minimal-test-caseb78sd-3318963475) message: Error from server (NotFound): pods \"some-pv\" not found"
time="2019-12-08T19:27:54Z" level=info msg="Step group node minimal-test-caseb78sd[0] (minimal-test-caseb78sd-3360048478) deemed failed: child 'minimal-test-caseb78sd-3318963475' failed" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:54Z" level=info msg="node minimal-test-caseb78sd[0] (minimal-test-caseb78sd-3360048478) phase Running -> Failed" namespace=argo-workflows workflow=minimal-test-caseb78sd            
time="2019-12-08T19:27:54Z" level=info msg="node minimal-test-caseb78sd[0] (minimal-test-caseb78sd-3360048478) message: child 'minimal-test-caseb78sd-3318963475' failed" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:54Z" level=info msg="node minimal-test-caseb78sd[0] (minimal-test-caseb78sd-3360048478) finished: 2019-12-08 19:27:54.757577542 +0000 UTC" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:54Z" level=info msg="step group minimal-test-caseb78sd-3360048478 was unsuccessful: child'minimal-test-caseb78sd-3318963475' failed" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:54Z" level=info msg="Outbound nodes of minimal-test-caseb78sd-3318963475 is [minimal-test-caseb78sd-3318963475]" namespace=argo-workflows workflow=minimal-test-caseb78sd       
time="2019-12-08T19:27:54Z" level=info msg="Outbound nodes of minimal-test-caseb78sd is [minimal-test-caseb78sd-3318963475]" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:54Z" level=info msg="node minimal-test-caseb78sd (minimal-test-caseb78sd) phase Running -> Failed" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:54Z" level=info msg="node minimal-test-caseb78sd (minimal-test-caseb78sd) message: child 'minimal-test-caseb78sd-3318963475' failed" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:54Z" level=info msg="node minimal-test-caseb78sd (minimal-test-caseb78sd) finished: 2019-12-08 19:27:54.761743556 +0000 UTC" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:54Z" level=info msg="Checking daemoned children of minimal-test-caseb78sd" namespace=argo-workflows workflow=minimal-test-caseb78sd                                             
time="2019-12-08T19:27:54Z" level=info msg="Updated phase Running -> Failed" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:54Z" level=info msg="Updated message  -> child 'minimal-test-caseb78sd-3318963475' failed" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:54Z" level=info msg="Marking workflow completed" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:54Z" level=info msg="Checking daemoned children of " namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:54Z" level=info msg="Workflow update successful" namespace=argo-workflows workflow=minimal-test-caseb78sd
time="2019-12-08T19:27:55Z" level=info msg="Labeled pod argo-workflows/minimal-test-caseb78sd-3318963475 completed"
time="2019-12-08T19:28:36Z" level=info msg="Alloc=6684 TotalAlloc=101781 Sys=70078 NumGC=1690 Goroutines=69"
W1208 19:28:42.840582       1 reflector.go:289] github.com/argoproj/argo/workflow/controller/config_controller.go:116: watch of *v1.ConfigMap ended with: too old resource version: 19574043 (19576346)
@sarabala1979
Copy link
Member

It should be a new flag/element under resource ignore-not-found : true and the default is false.
you are more welcome to create PR. Thanks for creating this issue.

@sarabala1979 sarabala1979 added the type/feature Feature request label Dec 9, 2019
@sarabala1979 sarabala1979 added this to the v2.5 milestone Dec 9, 2019
@dmayle dmayle mentioned this issue Dec 11, 2019
Closed
@alexec alexec modified the milestones: v2.5, v2.6 Jan 27, 2020
@nicorikken nicorikken mentioned this issue Jan 28, 2020
Closed
@alexec alexec removed this from the v2.6 milestone Feb 6, 2020
@dmayle
Copy link
Contributor Author

dmayle commented May 14, 2020

I've just created a PR #3037

@alexec
Copy link
Contributor

alexec commented Sep 29, 2020

FIxed.

@alexec alexec closed this as completed Sep 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dmayle dmayle

Labels
type/feature Feature request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dmayle @alexec @sarabala1979