argo lint fails with "invalid configuration: no configuration has been provided"

[Ark-kun]

Repro steps:

Checklist:

• I've included the version.
• I've included reproduction steps.
• I've included the workflow YAML.
• I've included the logs.

What happened:

argo lint workflow.yaml used to work in Argo v2.3.0, but it no longer works in Argo v.2.7.4

Error: time="2020-04-21T01:50:18Z" level=fatal msg="invalid configuration: no configuration has been provided"

What you expected to happen:

I expect argo lint to validate my workflow

How to reproduce it (as minimally and precisely as possible):

Download Argo CLI to some hermetic environment (e.g. Docker container under build). Try running argo lint workflow.yaml. Get error.

Anything else we need to know?:

I think argo lint might be trying to create Kubernetes client even though it's not needed for linting.

Environment:

• Argo version:

v2.7.4

---

Message from the maintainers:

If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[alexec]

Can you please attach logs argo --loglevel DEBUG lint ...

[alexec]

Can you also attach the output of kubectl cluster-info?

[Ark-kun]

Can you also attach the output of kubectl cluster-info?

There is no cluster. We run the Argo lint CLI program inside a hermetic Docker container. It has no access to anything.

[Ark-kun]

Try running the following inside some container (e.g. busybox)

wget https://github.com/argoproj/argo/releases/download/v2.4.3/argo-linux-amd64 -O argo243
chmod +x argo243
./argo243 lint wf.yaml
>>>Workflow manifests validated

wget https://github.com/argoproj/argo/releases/download/v2.5.2/argo-linux-amd64 -O argo252
chmod +x argo252
./argo252 lint wf.yaml
>>>FATA[0000] invalid configuration: no configuration has been provided

[alexec]

OK. I can now repro this by doing:

docker run -it docker/whalesay:latest sh
apt-get update
apt-get install -y curl
curl https://github.com/argoproj/argo/releases/download/v2.7.5/argo-linux-amd64 -o argo275 -L
chmod +x argo275
./argo275 --loglevel debug lint

Unfortunately, I'm not sure we can acceptably fix this. In v2.5.2 we introduced workflow templates. To lint a workflow that contains a template, we must read the template from the cluster, so we must have a connection to the cluster. This means a hermetically sealed container cannot lint workflows that contain templates.

There are a couple of options open to you today:

If you want to use workflow templates:

• Remove the linting and hope for the best.
• Allow the container a connection to the cluster.

If do you not want to use them:

• We could look into adding an --offline flag to the "argo" binary for this. We would do this if we got a number of up-votes.

[alexec]

@sarabala1979 @jessesuen FYI

[Ark-kun]

To lint a workflow that contains a template, we must read the template from the cluster, so we must have a connection to the cluster.

What if we make this connection lazy? This way the workflows that do not use external templates can still be linted.

[Ark-kun]

Just some thoughts about the design of external templates.
When faced with a need to separate templates from workflows, in Kubeflow Pipelines we chose to mainly reference the external templates by URL. URLs are universal resource locators. They point to the exact location where component/template can be accessed. And they are significantly easier to download than cluster-specific CRDs. In Argo, a pipeline can also be downloaded from a URL, but not the external templates. I see an inconsistency here. So, in KFP a pipeline can be downloaded from URL and all child components can also be downloaded from URLs. Pipeline is just a kind of component and can be referred and downloaded the same way.

[alexec]

To lint a workflow that contains a template, we must read the template from the cluster, so we must have a connection to the cluster.

What if we make this connection lazy? This way the workflows that do not use external templates can still be linted.

The request happens pretty much as the CLI starts. This is pretty fundamental to the design of the API client - which is is why is it is probably impractical to fix.

But I'm not sure this is what you need - you need a syntax check - I think we can add on --offline flag to argo lint which does basic validation.

[Ark-kun]

But I'm not sure this is what you need - you need a syntax check - I think we can add on --offline flag to argo lint which does basic validation.

This sounds great. Thank you.

[alexec]

Cool. If you fancy helping - please take a look at the linked PR.

[danxmoran]

I see the linked PR got shelved, but chiming in: An --offline flag for argo lint is exactly what my team needs. Ideally it would still check the calls between all the WorkflowTemplate objects found in the inputs, and just ignore any "dangling" references.

If the Argo team is still willing to accept the feature but isn't interested in building it themselves, I'd be happy to try extending the closed PR on my own! Have been looking for a way to contribute some code.

[alexec]

See #3027

[alexec]

See #3027

[danxmoran]

The YAML editor from the linked issue would be awesome, but I think there is still a use-case for argo lint in a vacuum. For example, my team uses Helm to package & deploy WorkflowTemplate and CronWorkflow objects, using templating to inject global variables and DRY up repeated blocks of configuration. As part of CI, we'd like to be able to verify that the output of rendering a chart using default/example values produces valid Argo object definitions.

[alexec]

@danxmoran would you like to take over this work?

[danxmoran]

Sure! Do you think I should build off of your original (closed) PR, or should I start fresh?

[Montana]

Seems like I'm still having same issue, via:

curl -sLO https://github.com/argoproj/argo/releases/download/v2.12.2/argo-darwin-amd64.gz
chmod u+x argo2122
./argo2122 lint wf.yaml
>>>Workflow manifests validated

I'll try and run a:

--loglevel debug lint

Via I'm getting this error (which is the same @Ark-kun was getting, but it could be a config problem on my end), I've checked thoroughly, and don't see any, but if I do find it - I'll be sure to post it.

I get:

>>>FATA[0000] invalid configuration: no configuration has been provided

Odd, as it was working this morning. I might do a fresh install and see if this fixes the problem.

Thanks
-Montana Mendy

[alexec]

There is now more detailed help for configuring the CLI, please check it out:

https://argoproj.github.io/argo/cli/argo/

[Montana]

Thank you @alexec! I really appreciate it.

[StefanoFioravanzo]

@alexec I got a bit lost in the discussion and the several linked issues. Is there a way to argo lint a workflow yaml in an offline mode or not?

[mildewey]

This is still a PROBLEM. Running an argo server is not acceptable for something that's supposed to be linting. It's not a unit test. It's not a dry run. It's not supposed to know if the parameters are passed in right. It's not supposed to know about other workflows and whether or not they exist.

It just needs to lint it and make sure that no one did something like:

inputs:
  parameters:
     parameters:
     ...

(real live example that I'm trying to stop from ever happening again)

This is what everyone is asking for. Not a "would this work if I launched it now?" but "did I screw up my yaml on accident?" or "did I misinterpret the docs at some point?"

[alexec]

I think it is reasonable to be able to lint offline. I think you need to implement some API. See linked PR?

[StefanoFioravanzo]

@alexec What PR are you referring to? There are several linked to this issue. Can you just confirm this: is there or is there not a way to lint offline, or some workaround to achieve that?

[alexec]

You can lint offline, as long as you do not use template references.

[alexec]

There are three PRs linked to this issue that contain POCs for improving/generalizing linting. It's not a priority for the core team right now (we're focused on v3.0) so it'd be awesome if someone could pick them up.

[NikeNano]

How should the argo cli be used to lint offline @alexec the following example still fails:

docker run -it docker/whalesay:latest sh
apt-get update
apt-get install -y curl

curl -sLO https://github.com/argoproj/argo/releases/download/v2.12.10/argo-linux-amd64.gz
gunzip argo-linux-amd64.gz
chmod +x argo-linux-amd64
mv ./argo-linux-amd64 /usr/local/bin/argo
argo version
argo --loglevel debug lint

with

FATA[2021-03-15T19:27:11.899Z] invalid configuration: no configuration has been provided

related to: kubeflow/pipelines#4804

did you get it to work @StefanoFioravanzo?

[alexec]

Linked to this issue is a PoC for "offline linting". Would anyone like to complete the work?

#2848

[NikeNano]

Based upon your comment @alec in #2793 (comment)

You can lint offline, as long as you do not use template references.

I though this was already possible?

Linked to this issue is a PoC for "offline linting". Would anyone like to complete the work?

#2848

I will take a look on it.

[Bobgy]

I did some further investigation

My observed behavior of argo cli v2.12.10 is that:

• when kubeconfig is not available, argo lint fails with "invalid configuration: no configuration has been provided"
• when kubeconfig is available, but not connected to network, argo lint successfully lints local workflows. As long as those workflows do not use WorkflowTemplate resources etc.

For KFP's use-case, workflow specs will never use WorkflowTemplate, so it'd be good enough for us, as long as argo CLI can skip (or ignore error from) trying to load kubeconfig part. Do we need a separate offline command to achieve this goal?

[NikeNano]

This is my understanding as well:

• when kubeconfig is not available, argo lint fails with "invalid configuration: no configuration has been provided"
• when kubeconfig is available, but not connected to network, argo lint successfully lints local workflows. As long as those workflows do not use WorkflowTemplate resources etc.

Can we expect all users to have a kubeconfig if they have argo installed? I think it is resonable but maybe we should check for it or add the requirement to the error message.

[Bobgy]

Can we expect all users to have a kubeconfig if they have argo installed?

Maybe not, data scientists who only use KFP python SDK might install argo CLI for linting, but do not have direct access to a cluster.

[Bobgy]

After reading #2848, it seems most of the major changes are still necessary, so it's not a trivial change. A lot of code assumed access to server is possible.

[minglei-pony]

Is it possible to lint templates offline?

$ argo --offline template lint
FATA[2022-06-03T12:18:45.892Z] Not implemented for offline client, only valid for kind '--kinds=workflows'