Upgrading Test::JSON::Schema::Acceptance from 1.005 to 1.006 causes t/draft2019-09-acceptance.t to fail

[Ovid]

• JSON::Validator version: 4.16
• Perl version: 5.26.2
• Operating system: MacOS 11.2.2 and Debian (unknown)

Because this is caused by the upgrading of a module not in this distribution, I reported the error in that module. However, Karen states that the code did not change between 1.005 and 1.006 and asked that I report the error here.

Steps to reproduce the behavior

Run this shell script:

#!/usr/bin/env bash

set -x

git clone git@github.com:jhthorsen/json-validator.git
cd json-validator

cpanm https://cpan.metacpan.org/authors/id/E/ET/ETHER/Test-JSON-Schema-Acceptance-1.006.tar.gz
TEST_ACCEPTANCE=1 prove -l t/draft2019-09-acceptance.t

cpanm https://cpan.metacpan.org/authors/id/E/ET/ETHER/Test-JSON-Schema-Acceptance-1.005.tar.gz
TEST_ACCEPTANCE=1 prove -l t/draft2019-09-acceptance.t

And you get the following output (notice that the test fails for the 1.006 and passes for 1.005):

$ ./test.sh
+ git clone git@github.com:jhthorsen/json-validator.git
Cloning into 'json-validator'...
remote: Enumerating objects: 102, done.
remote: Counting objects: 100% (102/102), done.
remote: Compressing objects: 100% (64/64), done.
remote: Total 8112 (delta 46), reused 70 (delta 27), pack-reused 8010
Receiving objects: 100% (8112/8112), 3.80 MiB | 4.67 MiB/s, done.
Resolving deltas: 100% (4751/4751), done.
+ cd json-validator
+ cpanm https://cpan.metacpan.org/authors/id/E/ET/ETHER/Test-JSON-Schema-Acceptance-1.006.tar.gz
--> Working on https://cpan.metacpan.org/authors/id/E/ET/ETHER/Test-JSON-Schema-Acceptance-1.006.tar.gz
Fetching https://cpan.metacpan.org/authors/id/E/ET/ETHER/Test-JSON-Schema-Acceptance-1.006.tar.gz ... OK
Configuring Test-JSON-Schema-Acceptance-1.006 ... OK
Building and testing Test-JSON-Schema-Acceptance-1.006 ... OK
Successfully installed Test-JSON-Schema-Acceptance-1.006
1 distribution installed
+ TEST_ACCEPTANCE=1
+ prove -l t/draft2019-09-acceptance.t
t/draft2019-09-acceptance.t .. 144/?     # Failed test 'test failed'
    # at t/Helper.pm line 47.
    # expected false; got true
# Failed test 'defs.json: "validate definition against metaschema" - "invalid definition schema"'
# at t/Helper.pm line 47.
t/draft2019-09-acceptance.t .. 944/? # Looks like you failed 1 test of 987.
t/draft2019-09-acceptance.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/987 subtests
        (136 TODO tests unexpectedly succeeded)

Test Summary Report
-------------------
t/draft2019-09-acceptance.t (Wstat: 256 Tests: 987 Failed: 1)
  Failed test:  192
  TODO passed:   63-64, 145-148, 660-691, 712, 723, 726-727
                830-923
  Non-zero exit status: 1
Files=1, Tests=987, 20 wallclock secs ( 0.20 usr  0.02 sys + 18.12 cusr  0.67 csys = 19.01 CPU)
Result: FAIL
+ cpanm https://cpan.metacpan.org/authors/id/E/ET/ETHER/Test-JSON-Schema-Acceptance-1.005.tar.gz
--> Working on https://cpan.metacpan.org/authors/id/E/ET/ETHER/Test-JSON-Schema-Acceptance-1.005.tar.gz
Fetching https://cpan.metacpan.org/authors/id/E/ET/ETHER/Test-JSON-Schema-Acceptance-1.005.tar.gz ... OK
Configuring Test-JSON-Schema-Acceptance-1.005 ... OK
Building and testing Test-JSON-Schema-Acceptance-1.005 ... OK
Successfully installed Test-JSON-Schema-Acceptance-1.005
1 distribution installed
+ TEST_ACCEPTANCE=1
+ prove -l t/draft2019-09-acceptance.t
t/draft2019-09-acceptance.t .. ok
All tests successful.

Test Summary Report
-------------------
t/draft2019-09-acceptance.t (Wstat: 0 Tests: 977 Failed: 0)
  TODO passed:   63-64, 145-148, 192, 660-691, 712, 723
                726-727, 830-913
Files=1, Tests=977, 20 wallclock secs ( 0.20 usr  0.01 sys + 18.14 cusr  0.64 csys = 18.99 CPU)
Result: PASS

Expected behavior

After upgrading Test::JSON::Schema::Acceptance from 1.005 to 1.006, I expect t/draft2019-09-acceptance.t to pass

Actual behavior

After upgrading Test::JSON::Schema::Acceptance from 1.005 to 1.006,t/draft2019-09-acceptance.t failed with:

t/draft2019-09-acceptance.t .. 177/?     # Failed test 'test failed'
    # at t/Helper.pm line 47.
    # expected false; got true

# Failed test 'defs.json: "validate definition against metaschema" - "invalid definition schema"'

[jhthorsen]

Unfortunately this issue has been closed by skipping another test. I've created #247 so they are not forgotten though.