Build scan shows passing when xray fails

[jasonmcintosh]

[Info] Deploying build info...
[Info] Build info successfully deployed. Browse it in Artifactory under 
****
Scanning build for vulnerabilities
[Info] Triggered Xray build scan... The scan may take a few minutes.
[Info] Xray scan completed.
{
  "summary": {
    "total_alerts": 0,
    "fail_build": false,
    "message": "Build orca-armory number 0.0.1-dev.121.65a1d0f was scanned by Xray and passed with no Alerts",
    "more_details_url": "***"
  },
  "alerts": [],
  "licenses": [
    {
      "name": "Unknown",
      "full_name": "Unknown license",
      "components": [
        "build://orca-armory:0.0.1-dev.121.65a1d0f"
      ]
    }
  ]
}

However, looking at x-ray it shows that the build has a number of issues. Workflow steps (with most things masked):

 JFROG_ARGS: --url *** --user *** --password ***
docker build ***/$BASE_PROJECT:$BUILD_NUMBER
jfrog rt docker-push ***/$BASE_PROJECT:$BUILD_NUMBER docker-local --build-name=$BASE_PROJECT-armory --build-number=$BUILD_NUMBER $JFROG_ARGS
  jfrog rt build-publish $BASE_PROJECT $BUILD_NUMBER $JFROG_ARGS
  jfrog rt build-scan  $BASE_PROJECT-armory $BUILD_NUMBER $JFROG_ARGS

[yahavi]

Thanks for reporting this issue, @jasonmcintosh.
I transfer it to jfrog-cli repository for further investigation.
Just adding that the relevant JFrog CLI version is 1.33.1.
Can you please add your Xray version?
Thanks!

[eyalbe4]

@jasonmcintosh,
The reason why the build scan shows passing, is because JFrog CLI returned a positive (zero) exit code for the build-scan command. The reason for this is that Xray returned "fail_build": false in the response summary (see in the JSON response you shared above). You need to configure a Watch in Xray, so that it know when to fail the build. That will result in "fail_build": true and a non zero exit code.
Please let us know if this works for you.

[jasonmcintosh]

@eyalbe4 Umm, did you read the command args up there?

  jfrog rt docker-push REPO.jfrog.io/armory/$BASE_PROJECT:$BUILD_NUMBER docker-local --build-name=$BASE_PROJECT --build-number=$BUILD_NUMBER $JFROG_ARGS
  jfrog rt build-publish $BASE_PROJECT $BUILD_NUMBER $JFROG_ARGS
  jfrog rt build-scan $BASE_PROJECT $BUILD_NUMBER $JFROG_ARGS

Note on the build-scan. NO "--fail" option. Just tried this again and got similar behavior:

[Info] Xray scan completed.
{
  "summary": {
    "total_alerts": 0,
    "fail_build": false,
    "message": "Build *** number *** was scanned by Xray and passed with no Alerts",
    "more_details_url": "https://URL.jfrog.io/ui/builds/****/***"
  },
  "alerts": [],
  "licenses": []
}

Ran with:

      uses: jfrog/setup-jfrog-cli@v1
      with:
        version: 1.32.4

I checked the UI. Initially it showed no vulnerabilities, yet after about 15 seconds or so, I refreshed the page and then vulnerabilities DID show up. This means jfrog CLI is pasisng builds that SHOULD be failing. With NO CHANGES on artifactory or x-ray, i reran the build-scan and the 2nd run fails.

[eyalbe4]

@jasonmcintosh,
Is the summary returned by Xray correct? (see the response above). It shows "total_alerts": 0.

[jasonmcintosh]

Yeah it's incorrect - that's the FIRST run. I modified the github action to do a build scan, sleep 30 seconds, then trigger another build scan. Here's the output...

[Info] Xray scan completed.
{
  "summary": {
    "total_alerts": 0,
    "fail_build": false,
    "message": "Build *** number *** was scanned by Xray and passed with no Alerts",
    "more_details_url": "https://***.jfrog.io/ui/builds/***/****"
  },
  "alerts": [],
  "licenses": []
}
[Info] Triggered Xray build scan... The scan may take a few minutes.
[Info] Xray scan completed.
[Error] Build Scan Failed
{
  "summary": {
    "total_alerts": 1,
    "fail_build": true,
    "message": "Build **** number ***** was scanned by Xray and 1 Alerts were generated",
    "more_details_url": "https://****.jfrog.io/ui/builds/****/****"
  },
  "alerts": [
    {
      "created": "2020-03-06T07:14:45.91020166Z",
      "top_severity": "High",```

[jasonmcintosh]

Here's the relevant workflow steps:

          jfrog rt build-publish $BASE_PROJECT-armory $BUILD_NUMBER $JFROG_ARGS
          jfrog rt build-scan $BASE_PROJECT-armory $BUILD_NUMBER $JFROG_ARGS
          sleep 30
          jfrog rt build-scan $BASE_PROJECT-armory $BUILD_NUMBER $JFROG_ARGS