Need a way to configure exit codes

[andrew-glenn]

https://github.com/aws-cloudformation/cfn-python-lint/blob/0ad76abe7dbd33dc404cd1cedd85c9643822d347/src/cfnlint/core.py#L55-L66

I'm not finding a way to non-zero exit on Errors only. Currently, based on the logic above, an informational message will trigger a non-zero exit code (8). Working around this in CICD pipelines is frustrating. A way to say --non-zero-exit-code ERROR would be splendid.

[PatMyron]

Thought about the exit codes quite a bit. I'd love for informational level rules to be displayed in the output without affecting the exit code, potentially even as the default behavior

level   code
           0
E          2
W          4
I          8
E W        6
E I       10
W I       12
E W I     14

echo $? % 4 | bc    # exit code modulo 4 should filter warnings and informational
echo $? % 8 | bc    # exit code modulo 8 should filter informational

---

Can also generally workaround with || true:

infracost/infracost#1478 (comment)

---

@andrew-glenn if you don't care about even seeing the warning and informational violations, simply --ignore-checks W should completely ignore warnings

[andrew-glenn]

In AWS-Native services (ala CodeBuild, and CodePIpeline that leverages CodeBuild), the shell environment is by-default configured to exit on non-zero status codes.

There are things that can be done to mitigate cfn-lint's behavior, such as set +e, trapping and examining the exit code, and things of that nature. Candidly though, it shouldn't be necessary.

The current behavior puts me in a position where I can't trust the binary notion that non-zero == failure, and instead need to inspect the exit codes (via source-code, subject to change) to determine the path forward.

At minimum (5pm Friday, before Vacation minimum), I'd expect to see this documented in --help and the README.

[andrew-glenn]

@PatMyron - the documentation seems to indicate that's possible. I'm not finding that to be the case, however.

❯ cfn-lint --ignore-checks W templates/*
usage:
Basic: cfn-lint test.yaml
Ignore a rule: cfn-lint -i E3012 -- test.yaml
Configure a rule: cfn-lint -x E3012:strict=false -t test.yaml
Lint all yaml files in a folder: cfn-lint dir/**/*.yaml

CloudFormation Linter

(...)

❯ cfn-lint --version
cfn-lint 0.33.0

Nor does this...

❯ cfn-lint -l | egrep -i '^(I|W)[0-9]' | cut -d ':' -f 1 | tr '\n' ',' | sed -e 's/,$//g'
W1001,W1011,W1019,W1020,W2001,W2030,W2501,W2506,W2509,W2510,W2511,W2531,W3002,W3005,W3010,W3011,W4001,W6001,W7001,W8001
❯ WARNING_RULES=$(cfn-lint -l | egrep -i '^(I|W)[0-9]' | cut -d ':' -f 1 | tr '\n' ',' | sed -e 's/,$//g')
❯ cfn-lint --ignore-checks ${WARNING_RULES} templates/*

[PatMyron]

might just be the option order? think --template needs to be explicit if not the first option

try one of these:

cfn-lint templates/* --ignore-checks W
cfn-lint --ignore-checks W --template templates/*

[andrew-glenn]

Closing the gap - both of those suggestions work.

There's another issue open to document the exit codes. closing.

[kddejong]

I think the only trick here is if someone wants to see the messages but change the exit code. I'm glad the --ignore-checks W works in this case.

[aihaddad]

as @kddejong this is exactly the use case. Legacy code base that we're fixing gradually. I want to be able to see the warning so that I can go and fix them bit by bit, but I don't want it to exit with non-zero so that it breaks Jenkins. Just the way ESLint, for example, works

[kddejong]

Another way to write @PatMyron's example

cfn-lint --include-checks I -- template.yaml || (($? % 4 ? 0 : 1))

if exit code is 2 from cfn-lint this will return 1
if exit code is 4 from cfn-lint this will return 0
if exit code is 6 from cfn-lint this will return 1
if exit code is 8 from cfn-lint this will return 0
if exit code is 12 from cfn-lint this will return 0
if exit code is 14 from cfn-lint this will return 1

[kddejong]

Given the amount of issues related to this I believe bringing some of the workaround mechanisms into the cfn-lint code base make sense.

[kddejong]

created #2436 to help move this logic into cfn-lint

[Alex-Dettinger]

@kddejong #2346 is not related to this error. Is a fix for this actively being worked on?

I've seen a number of reported issues that I am also having where github actions treats warnings from cfn-lint as an error and fails since version v0.66.0. The only viable workaround is to disable warnings or the specific warning you have trouble with, which defeats the purpose of warnings. Suppressing warnings is not a fix to this problem.

#2297
#897

[kddejong]

Wrong #. Updated now. The workarounds above will work for you and release v0.90.0 has a new way to configure exit codes.

You can look at #2460 to see the readme updates on how to use it.