Testing hysteresis with different selection methods

[greenc-FNAL]

Since 4beb9fc, develop has exhibited the following worrying behavior:

spack test -vv cmd_extensions.py

executes two tests, cmd_extensions.py::test_simple_command_extension and cmd_extensions.py::test_command_with_import, the former passing and the latter failing with a strange YAML validation schema error:

<snip>
self = <jsonschema.validators.Validator object at 0x7f4b72b7a0d0>
schema = {'$schema': 'http://json-schema.org/schema#', 'additionalProperties': False, 'properties': {'config': {'default': {}, ...{...}]}, 'ccache': {'type': 'boolean'}, ...}, 'type': 'object'}}, 'title': 'Spack core configuration file schema', ...}
types = (), resolver = None, format_checker = None

    def __init__(
        self, schema, types=(), resolver=None, format_checker=None,
    ):
        self._types = dict(self.DEFAULT_TYPES)
        self._types.update(types)

        if resolver is None:
>           resolver = RefResolver.from_schema(schema)
E           AttributeError: 'NoneType' object has no attribute 'from_schema'

../../external/jsonschema/validators.py:64: AttributeError

In contrast:

spack test -vvk cmd_extensions

while executing the same two tests—in apparently the same order—shows that both tests pass. This is in fact what happens to these tests in CI.

Any insight into locating the source of the hysteresis would be appreciated.

[greenc-FNAL]

@tgamblin

[tgamblin]

@chissg: see #14319 for details on the change. You can add a -k option to get the old behavior you want.

I agree that pytest's error message is unhelpful here... I'd normally say we should upgrade the external package, but we're tied to a relatively old version of pytest to continue supporting Python 2.6. Would it help if we added something to catch the attribute error and say "did you mean..."?

[greenc-FNAL]

@tgamblin I'm still not sure of the correlation between the first form of the invocation and the error. What is pytest not doing that is causing the second test to fail, or is there a weakness in the test I should be fixing?

[tldahlgren]

FWIW. I ran the tests both ways -- from develop at 6b3e173 -- and they pass regardless.

[greenc-FNAL]

Still trying to ascertain what makes this reproducible for me but not for anyone else. Platform is SLF7.7.

[greenc-FNAL]

@tgamblin I have been able to reproduce this on a (different) SLF7.7 machine. The error does not occur if /usr/bin/python3 is available, which is used for preference. It does occur if /usr/bin/python (corresponding to version 2.7.5) is invoked. On the machine where python3 is not present the invocation of spack test -vv cmd_extensions.py fails reliably; on the other machine it passes unless I change the order of preference of Python executables at

spack/bin/spack

Line 14 in 673a725

for cmd in python3 python python2; do

to invoke Python2 for preference. Quite what is sensitive to Python2 versus Python3 I have no idea, but at least I know I'm not hallucinating.

[alalazo]

@greenc-FNAL Can we close this one, since Spack doesn't support running with Python 2 anymore?