Validate partially fails regression test on issue 388

[jpl-jengelke]

🐛 Describe the bug identified during I&T

Regression test validate#388 partially fails in the regression testing tool. It appears that a test report describing PDF validation errors, as required by the Build 12.1 Test Plan, is no longer generated.

Posted here for discussion or to address root cause. Has the requirement for the report been eliminated in subsequent design/development changes? If YES then we should change the test executionto eliminate this requirement.

🥼 Related Test Case(s)

B12.1, VAL.1

🔁 : Related issues

#388

---

➕ Additional Details

https://pds-jenkins.jpl.nasa.gov/job/console-regression-tests/job/develop/306/console
https://cae-testrail.jpl.nasa.gov/testrail/index.php?/tests/view/4072906

📜 To Reproduce

See steps here
See https://pds-engineering.jpl.nasa.gov/file/build12.1testprocs_v10.pdf

🕵️ Expected behavior

A report is generated during a failure. (If this is no longer the case due to subsequent design/development changes, then this bug should be closed and the test modified to accommodate the changes.

📚 Version of Software Used

Validate 3.0.3

🩺 Test Data / Additional context

https://github.jpl.nasa.gov/PDSEN/console-test-tool/tree/develop/datasets/validate/388/testdata

🏞Screenshots

🖥 System Info

• OS: Mac OS X 11.7.1 or Ubuntu-latest (Dockerized)
• java 11.0.3 2019-04-16 LTS
• Validate 2.4.0 or 3.0.0
• GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin20)

---

🦄 Related requirements

⚙️ Engineering Details

[jordanpadams]

*********************************** COMMAND  ***********************************
validate -R pds4.label -t /data/jenkins/workspace/console-regression-
tests_develop/datasets/validate/388/testdata/pixl_edr_sis_bad.xml -v2

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/data/jenkins/workspace/console-regression-tests_develop/venv/bin:/data/jenkins/workspace/console-regression-tests_develop/test-apps/validate/validate-3.0.3/bin
********************************* TEST RESULT  ********************************* 
PDS Validate Tool Report

Configuration:
   Version                       3.0.3
   Date                          2022-11-14T18:31:55Z

Parameters:
   Targets                       [file:/data/jenkins/workspace/console-regression-tests_develop/datasets/validate/388/testdata/pixl_edr_sis_bad.xml]
   Rule Type                     pds4.label
   Severity Level                WARNING
   Recurse Directories           true
   File Filters Used             [*.xml, *.XML]
   Data Content Validation       on
   Product Level Validation      on
   Max Errors                    100000
   Registered Contexts File      /data/jenkins/workspace/console-regression-tests_develop/test-apps/validate/validate-3.0.3/resources/registered_context_products.json



Product Level Validation Results

  FAIL: file:/data/jenkins/workspace/console-regression-tests_develop/datasets/validate/388/testdata/pixl_edr_sis_bad.xml
      ERROR  [error.pdf.file.not_pdfa_compliant]   Invalid PDF/A version detected for file:/data/jenkins/workspace/console-regression-tests_develop/datasets/validate/388/testdata/pixl_edr_sis_bad.pdf. Expected: 1a or 1b. Actual: 2b
        1 product validation(s) completed

Summary:

  1 error(s)
  0 warning(s)

  Product Validation Summary:
    0          product(s) passed
    1          product(s) failed
    0          product(s) skipped

  Referential Integrity Check Summary:
    0          check(s) passed
    0          check(s) failed
    0          check(s) skipped

  Message Types:
    1            error.pdf.file.not_pdfa_compliant

[jpl-jengelke]

Correct @jordanpadams . In the earlier tests it was expecting an extra pdf-specific report file to be created. The file (failure report) is not there now. Is that OK?

[jpl-jengelke]

I'll re-review this ticket and provide more detail. I located a better test set. Pending. ...

[nutjob4life]

FYI, @jpl-jengelke and I had a lengthy Slack discussion about this, hence his comment ↑.

For my own memory, this is the command I ran:

validate --report-file blah.json --report-style json --target 388/pixl_edr_sis_bad.xml

I repeated it for various versions of validate going back through the 2 series and into the 1 series but never got a second PDF-specific report file. I hope John can figure out whither or thither it should be! 😁

[jpl-jengelke]

The original test had to be adapted to regression test requirements. The original test data here for issue 388 was not appropriate for Vera report generation since the accompanying PDF was well formed. When validate issued its previous analysis that the PDF was the incorrect type, that was all that it should have generated. To yield a report, a malformed PDF file had to be used. So known malformed test data from #328 was passed through validate which then generated a definitive Vera report documenting the known errors. Using this an additional test execution was added to the regression to correctly verify report generation.

Closing this as not a bug.