bad character range while listing or running refstack tests

[chkumar246]

Issue description

Download the refstack tests:

wget "https://refstack.openstack.org/api/v1/guidelines/2018.02/tests?target=platform&type=required&alias=true&flag=false" -O 2018.02-test-list.txt

then list or run refstack tests using stestr:

(.venv) [chkumar246@fedora stestr]$ stestr list --whitelist-file 2018.02-test-list.txt 
bad character range
(.venv) [chkumar246@fedora stestr]$ stestr list --whitelist-file 2018.02-test-list.txt --debug
bad character range
Traceback (most recent call last):
  File "/home/chkumar246/arena/stackers/stestr/.venv/lib/python2.7/site-packages/cliff/app.py", line 399, in run_subcommand
    result = cmd.run(parsed_args)
  File "/home/chkumar246/arena/stackers/stestr/.venv/lib/python2.7/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/home/chkumar246/arena/stackers/stestr/stestr/commands/list.py", line 72, in take_action
    filters=filters)
  File "/home/chkumar246/arena/stackers/stestr/stestr/commands/list.py", line 121, in list_command
    cmd.setUp()
  File "/home/chkumar246/arena/stackers/stestr/stestr/test_processor.py", line 148, in setUp
    black_regex=self.black_regex)
  File "/home/chkumar246/arena/stackers/stestr/stestr/selection.py", line 113, in construct_list
    list_of_test_cases = filter_tests(regexes, test_ids)
  File "/home/chkumar246/arena/stackers/stestr/stestr/selection.py", line 29, in filter_tests
    _filters = list(map(re.compile, filters))
  File "/home/chkumar246/arena/stackers/stestr/.venv/lib64/python2.7/re.py", line 194, in compile
    return _compile(pattern, flags)
  File "/home/chkumar246/arena/stackers/stestr/.venv/lib64/python2.7/re.py", line 251, in _compile
    raise error, v # invalid expression
error: bad character range
Traceback (most recent call last):
  File "/home/chkumar246/arena/stackers/stestr/.venv/bin/stestr", line 10, in <module>
    sys.exit(main())
  File "/home/chkumar246/arena/stackers/stestr/stestr/cli.py", line 101, in main
    return cli.run(argv)
  File "/home/chkumar246/arena/stackers/stestr/.venv/lib/python2.7/site-packages/cliff/app.py", line 278, in run
    result = self.run_subcommand(remainder)
  File "/home/chkumar246/arena/stackers/stestr/.venv/lib/python2.7/site-packages/cliff/app.py", line 399, in run_subcommand
    result = cmd.run(parsed_args)
  File "/home/chkumar246/arena/stackers/stestr/.venv/lib/python2.7/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/home/chkumar246/arena/stackers/stestr/stestr/commands/list.py", line 72, in take_action
    filters=filters)
  File "/home/chkumar246/arena/stackers/stestr/stestr/commands/list.py", line 121, in list_command
    cmd.setUp()
  File "/home/chkumar246/arena/stackers/stestr/stestr/test_processor.py", line 148, in setUp
    black_regex=self.black_regex)
  File "/home/chkumar246/arena/stackers/stestr/stestr/selection.py", line 113, in construct_list
    list_of_test_cases = filter_tests(regexes, test_ids)
  File "/home/chkumar246/arena/stackers/stestr/stestr/selection.py", line 29, in filter_tests
    _filters = list(map(re.compile, filters))
  File "/home/chkumar246/arena/stackers/stestr/.venv/lib64/python2.7/re.py", line 194, in compile
    return _compile(pattern, flags)
  File "/home/chkumar246/arena/stackers/stestr/.venv/lib64/python2.7/re.py", line 251, in _compile
    raise error, v # invalid expression
sre_constants.error: bad character range

When I tried to run the same with os-testr it works fine:

(py27) [chkumar246@fedora tempest]$ ostestr -l --whitelist-file 2018.02-test-list.txt
tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_create_agent[id-1fc6bdc8-0b6d-4cc7-9f30-9b04fabe5b90]
tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_delete_agent[id-470e0b89-386f-407b-91fd-819737d0b335]
tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents[id-6a326c69-654b-438a-80a3-34bcc454e138]
tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents_with_filter[id-eabadde4-3cd7-4ec4-a4b5-5a936d2d4408]
tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_update_agent[id-dc9ffd51-1c50-4f0e-a820-ae6d2a568a9e]

Expected behavior and actual behavior

stestr would able to list the tests.

I think there is something fishy going on here: https://github.com/mtreinish/stestr/blob/master/stestr/selection.py#L29

System information

stestr version (stestr --version):
stestr 2.1.1.dev4
Python release (python --version):
Python 2.7.15
pip packages (pip freeze):
alabaster==0.7.10
alembic==0.9.9
Babel==2.6.0
certifi==2018.4.16
chardet==3.0.4
cliff==2.12.0
cmd2==0.8.7
contextlib2==0.5.5
coverage==4.5.1
ddt==1.1.3
debtcollector==1.19.0
decorator==4.3.0
docutils==0.14
enum34==1.1.6
extras==1.0.0
fixtures==3.0.0
flake8==2.5.5
funcsigs==1.0.2
future==0.16.0
hacking==0.11.0
idna==2.7
imagesize==1.0.0
iso8601==0.1.12
Jinja2==2.10
linecache2==1.0.0
Mako==1.0.7
MarkupSafe==1.0
mccabe==0.2.1
mock==2.0.0
monotonic==1.5
netaddr==0.7.19
netifaces==0.10.7
oslo.config==6.2.1
oslo.db==4.38.0
oslo.i18n==3.20.0
oslo.utils==3.36.2
packaging==17.1
pbr==4.0.4
pep8==1.5.7
prettytable==0.7.2
pyflakes==0.8.1
Pygments==2.2.0
pyparsing==2.2.0
pyperclip==1.6.2
python-dateutil==2.7.3
python-editor==1.0.3
python-mimeparse==1.6.0
python-subunit==1.3.0
pytz==2018.4
PyYAML==3.12
requests==2.19.1
rfc3986==1.1.0
six==1.11.0
snowballstemmer==1.2.1
Sphinx==1.7.5
sphinxcontrib-websupport==1.1.0
SQLAlchemy==1.2.8
sqlalchemy-migrate==0.11.0
sqlparse==0.2.4
-e git+https://github.com/mtreinish/stestr.git@113015c2c7b094cdfad72198fc76c9de1b72ff97#egg=stestr
stevedore==1.28.0
subprocess32==3.5.2
subunit2sql==1.9.0
Tempita==0.5.2
testresources==2.0.1
testscenarios==0.5.0
testtools==2.3.0
traceback2==1.4.0
typing==3.6.4
unicodecsv==0.14.1
unittest2==1.1.0
urllib3==1.23
voluptuous==0.11.1
wcwidth==0.1.7
wrapt==1.10.11

Operating System:

Fedora 28

[masayukig]

I think it causes because the file has idempotent ids, and it's like regexes.

tempest.api.volume.test_volumes_list.VolumesListTestJSON.test_volumes_list_by_bootable[id-2016a942-3020-40d7-95ce-7613bf8407ce]

And ostestr just ignores the --whitelist-file option in the list mode.

So, you would face the same error if you run ostestr without -l option like below.

$ ostestr  --whitelist-file ./2018.02-test-list.txt
Traceback (most recent call last):
  File "/home/masayuki/git/openstack/os-testr/.tox/py36/bin/ostestr", line 11, in <module>
    sys.exit(main())
  File "/home/masayuki/git/openstack/os-testr/.tox/py36/lib/python3.6/site-packages/os_testr/ostestr.py", line 273, in main
    exit(ostestr(sys.argv[1:]))
  File "/home/masayuki/git/openstack/os-testr/.tox/py36/lib/python3.6/site-packages/os_testr/ostestr.py", line 269, in ostestr
    return _select_and_call_runner(opts, regex, others)
  File "/home/masayuki/git/openstack/os-testr/.tox/py36/lib/python3.6/site-packages/os_testr/ostestr.py", line 222, in _select_and_call_runner
    black_regex=opts.black_regex)
  File "/home/masayuki/git/openstack/os-testr/.tox/py36/lib/python3.6/site-packages/os_testr/ostestr.py", line 177, in call_testr
    black_regex=black_regex)
  File "/home/masayuki/git/openstack/os-testr/.tox/py36/lib/python3.6/site-packages/stestr/commands/run.py", line 416, in run_command
    suppress_attachments=suppress_attachments)
  File "/home/masayuki/git/openstack/os-testr/.tox/py36/lib/python3.6/site-packages/stestr/commands/run.py", line 463, in _run_tests
    cmd.setUp()
  File "/home/masayuki/git/openstack/os-testr/.tox/py36/lib/python3.6/site-packages/stestr/test_processor.py", line 148, in setUp
    black_regex=self.black_regex)
  File "/home/masayuki/git/openstack/os-testr/.tox/py36/lib/python3.6/site-packages/stestr/selection.py", line 113, in construct_list
    list_of_test_cases = filter_tests(regexes, test_ids)
  File "/home/masayuki/git/openstack/os-testr/.tox/py36/lib/python3.6/site-packages/stestr/selection.py", line 29, in filter_tests
    _filters = list(map(re.compile, filters))
  File "/home/masayuki/git/openstack/os-testr/.tox/py36/lib64/python3.6/re.py", line 233, in compile
    return _compile(pattern, flags)
  File "/home/masayuki/git/openstack/os-testr/.tox/py36/lib64/python3.6/re.py", line 301, in _compile
    p = sre_compile.compile(pattern, flags)
  File "/home/masayuki/git/openstack/os-testr/.tox/py36/lib64/python3.6/sre_compile.py", line 562, in compile
    p = sre_parse.parse(p, flags)
  File "/home/masayuki/git/openstack/os-testr/.tox/py36/lib64/python3.6/sre_parse.py", line 855, in parse
    p = _parse_sub(source, pattern, flags & SRE_FLAG_VERBOSE, 0)
  File "/home/masayuki/git/openstack/os-testr/.tox/py36/lib64/python3.6/sre_parse.py", line 416, in _parse_sub
    not nested and not items))
  File "/home/masayuki/git/openstack/os-testr/.tox/py36/lib64/python3.6/sre_parse.py", line 553, in _parse
    raise source.error(msg, len(this) + 1 + len(that))
sre_constants.error: bad character range 5-4 at position 92

[chkumar246]

@masayukig Thanks for the input, yup the same error is coming on ostestr also. Is it possible to add idempotent ids support in stestr also?

[masayukig]

I think it might be good if stestr run accepts stestr list output with idempotent ids like Tempest test list. But I think it might be a bit tricky change and a hard to keep consistency to distinguish regexes or idempotent ids.

@mtreinish Any thoughts?

[masayukig]

@chkumar246 actually, there can be not only idempotent ids but also decorators such as slow, negative, compute, volume, etc...

So, it might be possible with applying a filter which is using [0] after splitting with [ character.
But I think we need to introduce a new option for that like --whitelist-file-noregex.. (I'm not a fan of like this option addition, though..)

[masayukig]

Ah, I think stestr run --load-list is the option for that. But it's only in run not in list. So, one idea is enabling that bringing the option to list too.

Another idea is introducing --dry-run option to stestr run. But I feel that would be conflicted with stestr list command, though.

[mtreinish]

Right, this is not a bug. The docs for --whiteliest-file clearly say (in multiple places) that each line is treated as a separate regex. Passing an invalid regex in (which is what you're doing with that file from refstack) is things failing because of bad input. We maybe could have a better error message which doesn't result in a strack trace, but tha's a separate thing. This has nothing to do with idempotent ids or any other attributes. The '[' ']' surrounding the args are really just an artifact of displaying the attributes set on a test. You can already easily just filter on them using a valid regex.

What you have there is list for the --load-list argument on stestr run. Since you're specifying exactly what tests to run. IIRC the refstack/interop guideline docs actually explicitly say use --load-list.

@masayukig there isn't really a need for --load-list as an option on stestr list because there is nothing to list with --load-list, it would be the same as running cat load_list.txt. --load-list takes the place of discovery and any test selection and explicitly tells stestr run what the tests it needs to schedule and execute are.

[masayukig]

@mtreinish yeah, I noticed that. So, I was thinking --dry-run might be one option. But I feel it's a bit overkill/too much. So, I think your #182 change is enough.