--failon command does not return status 1

[SFDX-Sam]

Good evening! I've implemented flow scanner on both a gitlab pipeline and my local windows machine, and I cannot get the scanner to return a status code of 1 when using the --failon command.

I run the following command:
sf flow:scan --failon error --config "scanner-config.json"

And it returns:

Identified 1 flows to scan... Scan complete
== DML Inside Loop Flow ==
Flow type: AutoLaunchedFlow

ERROR APIVersion
Details: 56
Newer API components may cause older versions of Flows to start behaving incorrectly due to differences in the underlying 
mechanics. The Api Version has been available as an attribute on the Flow since API v50.0 and it is recommended to limit 
variation and to update them on a regular basis.

ERROR FlowName
Details: The name DML_Inside_Loop_Flow does not meet the regex convention ===58
Readability of a flow is very important. Setting a naming convention for the Flow Name will improve the 
findability/searchability and overall consistency. It is recommended to at least provide a domain and a short description of 
the actions undertaken in the flow, in example Service_OrderFulfillment.


=== A total of 2 errors have been found in 1 flows.`

But does not seem to throw a status code of 1 - I have tried reducing the loglevel down to warning but still no luck.

[SFDX-Sam]

For reference - my gitlab pipeline looks like this:

####################################################
# The docker image the jobs initialize with.
# We use nodejs, so a node image makes sense.
# https://docs.gitlab.com/ee/ci/yaml/README.html#image
####################################################
image: "node:latest"

####################################################
# The sequential stages of this pipeline.
# Jobs within a stage run in parallel.
# https://docs.gitlab.com/ee/ci/yaml/README.html#stages
####################################################
stages:
  - metadata_scan
  - apex_scan

####################################################
# Metadata Scanning Phase
####################################################
metadata_scan:
  stage: metadata_scan
  allow_failure: false
  script:
    - install_salesforce_cli
    - install_lightning_flow_scanner
    - scan_metadata

  artifacts:
    paths:
      - metadata-scan-files/
    when: always


####################################################
# Helper Methods
####################################################

.sf_helpers: &sf_helpers |
  function install_salesforce_cli() {

    # Salesforce CLI Environment Variables
    # https://developer.salesforce.com/docs/atlas.en-us.sfdx_dev.meta/sfdx_dev/sfdx_dev_cli_env_variables.htm

    # By default, the CLI periodically checks for and installs updates.
    # Disable (false) this auto-update check to improve performance of CLI commands.
    export SF_AUTOUPDATE_DISABLE=false

    # Set to true if you want to use the generic UNIX keychain instead of the Linux libsecret library or macOS keychain.
    # Specify this variable when using the CLI with ssh or "headless" in a CI environment.
    export SF_USE_GENERIC_UNIX_KEYCHAIN=true

    # For force:package:version:create, disables automatic updates to the sfdx-project.json file.
    export SF_PROJECT_AUTOUPDATE_DISABLE_FOR_PACKAGE_VERSION_CREATE=true

    # Install Salesforce CLI
    npm install @salesforce/cli --global

    # Output CLI version and plug-in information
    sf update
    sf --version
    sf plugins --core

  }
  #Install the flow scanner plugin - it is not digitally signed so we need to echo y to accept the prompt.
  function install_lightning_flow_scanner() {
    echo 'y' | sf plugins:install lightning-flow-scanner
  }

  # Function to run the metadata scan on the current branch
  function scan_metadata() {
    local OUTPUT_DIRECTORY=metadata-scan-files
    mkdir -p $OUTPUT_DIRECTORY
    sf flow:scan --failon error --config "scanner-config.json" 

  }

[RubenHalman]

thank you for reporting this. First of all, I think it would be -failon "error" instead of -failon error.(not sure if that makes a difference) I am wondering however, if you don't use failing it is error level by default as per my understanding. so in this case you don't need the argument, although it should work ofcoùrse. what are you trying to achieve with the failon argument specifically?

[SFDX-Sam]

Hey @RubenHalman !

I think I just put it because the documentation didn't specify if it was required for throwing a non-zero status code or not.

It seems that even without the --failon flag, it still returns a status code of 1 (success).

I also tried -failon "error", but no dice, it did not give me any errors however, so I guess it is an accepted param.

[RubenHalman]

I am sorry. Can you show the --JSON output?

[SFDX-Sam]

Sure thing

{
  "status": 0,
  "result": {
    "summary": {
      "flowsNumber": 1,
      "errors": 2,
      "message": "A total of 2 errors have been found in 1 flows.",
      "errorLevelsDetails": {
        "error": 2
      }
    },
    "status": 1,
    "results": [
      {
        "flowName": "DML Inside Loop Flow",
        "ruleName": "APIVersion",
        "description": "Newer API components may cause older versions of Flows to start behaving incorrectly due to differences in the underlying mechanics. The Api Version has been available as an attribute on the Flow since API v50.0 and it is recommended to limit variation and to update them on a regular basis.",
        "severity": "error",
        "type": "flow",
        "details": "56"
      },
      {
        "flowName": "DML Inside Loop Flow",
        "ruleName": "FlowName",
        "description": "Readability of a flow is very important. Setting a naming convention for the Flow Name will improve the findability/searchability and overall consistency. It is recommended to at least provide a domain and a short description of the actions undertaken in the flow, in example Service_OrderFulfillment.",
        "severity": "error",
        "type": "flow",
        "details": "The name DML_Inside_Loop_Flow does not meet the regex convention ===58"
      }
    ]
  }
}

Interestingly, I can see the status here for each error, but it does not seem to exit the oclif process with the exit code - it seems possible based on a quick look here

[RubenHalman]

I probably need some input from @nvuillam here. Before v2, the core module threw an error resulting in an exit code, but it malformed the json. So I believe we changed it so the json is valid so it can be used for automation, but this is not really my expertise and requires a further deep dive

[nvuillam]

@RubenHalman @SFDX-Sam sorry for the delay (vacations ^^), this issue should be solved in #72 :)

[SFDX-Sam]

Thanks both!

[RubenHalman]

Hi @SFDX-Sam . Sorry for the delay and thank you again for your feedback. @nvuillam fix is included in the latest version 2.12. Please let us know if this sufficiently addresses your needs!