Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BF(TST): use sys.executable and not hardcoded "python" #5612

Merged
merged 2 commits into from Apr 29, 2021

Conversation

yarikoptic
Copy link
Member

@yarikoptic yarikoptic commented Apr 28, 2021

Detected while trying to build 0.14.2 for Debian

Lead to

======================================================================
ERROR: datalad.interface.tests.test_run_procedure.test_name_with_underscore
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
	self.test(*self.arg)
  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/tests/utils.py", line 578, in _wrap_with_tree
	return t(*(arg + (d,)), **kw)
  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/interface/tests/test_run_procedure.py", line 418, in test_name_with_underscore
	ds.run_procedure(spec=["print_args"])
  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/distribution/dataset.py", line 503, in apply_func
	return f(**kwargs)
  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/interface/utils.py", line 482, in eval_func
	return return_func(generator_func)(*args, **kwargs)
  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/interface/utils.py", line 470, in return_func
	results = list(results)
  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/interface/utils.py", line 389, in generator_func
	for r in _process_results(
  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/interface/utils.py", line 557, in _process_results
	for res in results:
  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/interface/run_procedure.py", line 448, in __call__
	for r in Run.__call__(
  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/interface/utils.py", line 389, in generator_func
	for r in _process_results(
  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/interface/utils.py", line 557, in _process_results
	for res in results:
  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/core/local/run.py", line 237, in __call__
	for r in run_command(cmd, dataset=dataset,
  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/core/local/run.py", line 685, in run_command
	raise exc
  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/core/local/run.py", line 464, in _execute_command
	runner.run(
  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/cmd.py", line 408, in run
	raise CommandError(
datalad.support.exceptions.CommandError: CommandError: 'python /tmp/datalad_temp_tree_test_name_with_underscoreydwd49na/.datalad/procedures/print_args' failed with exitcode 127 under /tmp/datalad_temp_tree_test_name_with_underscoreydwd49na

----------------------------------------------------------------------
Ran 1227 tests in 1944.566s

FAILED (SKIP=142, errors=1)
D: /a
D: /a/b/c
D: /a~
D: /a b/name
testing 0
testing 1
/tmp/datalad_temp_tree_test_paths_with_forward_slashest42h83pf
' |;&%b5{}\'" .datc '

Detected while trying to build 0.14.2 for Debian

Lead to

	======================================================================
	ERROR: datalad.interface.tests.test_run_procedure.test_name_with_underscore
	----------------------------------------------------------------------
	Traceback (most recent call last):
	  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
		self.test(*self.arg)
	  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/tests/utils.py", line 578, in _wrap_with_tree
		return t(*(arg + (d,)), **kw)
	  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/interface/tests/test_run_procedure.py", line 418, in test_name_with_underscore
		ds.run_procedure(spec=["print_args"])
	  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/distribution/dataset.py", line 503, in apply_func
		return f(**kwargs)
	  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/interface/utils.py", line 482, in eval_func
		return return_func(generator_func)(*args, **kwargs)
	  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/interface/utils.py", line 470, in return_func
		results = list(results)
	  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/interface/utils.py", line 389, in generator_func
		for r in _process_results(
	  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/interface/utils.py", line 557, in _process_results
		for res in results:
	  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/interface/run_procedure.py", line 448, in __call__
		for r in Run.__call__(
	  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/interface/utils.py", line 389, in generator_func
		for r in _process_results(
	  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/interface/utils.py", line 557, in _process_results
		for res in results:
	  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/core/local/run.py", line 237, in __call__
		for r in run_command(cmd, dataset=dataset,
	  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/core/local/run.py", line 685, in run_command
		raise exc
	  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/core/local/run.py", line 464, in _execute_command
		runner.run(
	  File "/build/datalad-0.14.2/.pybuild/cpython3_3.9_datalad/build/datalad/cmd.py", line 408, in run
		raise CommandError(
	datalad.support.exceptions.CommandError: CommandError: 'python /tmp/datalad_temp_tree_test_name_with_underscoreydwd49na/.datalad/procedures/print_args' failed with exitcode 127 under /tmp/datalad_temp_tree_test_name_with_underscoreydwd49na

	----------------------------------------------------------------------
	Ran 1227 tests in 1944.566s

	FAILED (SKIP=142, errors=1)
	D: /a
	D: /a/b/c
	D: /a~
	D: /a b/name
	testing 0
	testing 1
	/tmp/datalad_temp_tree_test_paths_with_forward_slashest42h83pf
	' |;&%b5{}\'" .datc '
@kyleam
Copy link
Contributor

kyleam commented Apr 29, 2021

The updated test is failing on windows:

[00:15:58] ======================================================================
[00:15:58] ERROR: datalad.interface.tests.test_run_procedure.test_name_with_underscore
[00:15:58] ----------------------------------------------------------------------
[00:15:58] Traceback (most recent call last):
[00:15:58]   File "C:\Python39-x64\lib\site-packages\nose\case.py", line 198, in runTest
[00:15:58]     self.test(*self.arg)
[00:15:58]   File "C:\Python39-x64\lib\site-packages\datalad\tests\utils.py", line 578, in _wrap_with_tree
[00:15:58]     return t(*(arg + (d,)), **kw)
[00:15:58]   File "C:\Python39-x64\lib\site-packages\datalad\interface\tests\test_run_procedure.py", line 418, in test_name_with_underscore
[00:15:58]     ds.run_procedure(spec=["print_args"])
[00:15:58]   File "C:\Python39-x64\lib\site-packages\datalad\distribution\dataset.py", line 503, in apply_func
[00:15:58]     return f(**kwargs)
[00:15:58]   File "C:\Python39-x64\lib\site-packages\datalad\interface\utils.py", line 486, in eval_func
[00:15:58]     return return_func(generator_func)(*args, **kwargs)
[00:15:58]   File "C:\Python39-x64\lib\site-packages\datalad\interface\utils.py", line 474, in return_func
[00:15:58]     results = list(results)
[00:15:58]   File "C:\Python39-x64\lib\site-packages\datalad\interface\utils.py", line 393, in generator_func
[00:15:58]     for r in _process_results(
[00:15:58]   File "C:\Python39-x64\lib\site-packages\datalad\interface\utils.py", line 561, in _process_results
[00:15:58]     for res in results:
[00:15:58]   File "C:\Python39-x64\lib\site-packages\datalad\interface\run_procedure.py", line 439, in __call__
[00:15:58]     raise ValueError("No idea how to execute procedure %s. "
[00:15:58] ValueError: No idea how to execute procedure C:\DLTMP\datalad_temp_tree__jmpbhkh\.datalad\procedures\print_args. Missing 'execute' permissions?

https://ci.appveyor.com/project/mih/datalad/builds/38916950/job/07ft7km2jucgn5q5#L814

@yarikoptic
Copy link
Member Author

Thanks for pasting. I will later troubleshoot in a VM.

…OVERRIDES_JSON

Needed on Windows when using sys.executable which would have a path
with backslashes.  Without proper serialization into json it would
fail de-serialize, and we would proceed while just issueing a warning
which is by default not even shown in CI runs.
@yarikoptic
Copy link
Member Author

well -- long story (did take some time to tune my fingers to windows again) short -- it was not a proper json in that test. \\ trips json.loads. And because we do not bother with logging while running tests we did not even see that warning.

$ DATALAD_LOG_LEVEL=WARNING python -m nose -s -v --pdb datalad/interface/tests/test_run_procedure.py:test_name_with_underscore
datalad.interface.tests.test_run_procedure.test_name_with_underscore ... [WARNING] Failed to load DATALAD_CONFIG_OVERRIDES_JSON: Invalid \escape: line 1 column 50 (char 49)
[WARNING] Failed to load DATALAD_CONFIG_OVERRIDES_JSON: Invalid \escape: line 1 column 50 (char 49)
> c:\users\datalad\pycharmprojects\datalad\datalad\interface\run_procedure.py(439)__call__()
-> raise ValueError("No idea how to execute procedure %s. "

Pushed a fix for this which would just use json.dumps.

IMHO it might be yet another case supporting our DX-costly (and possibly UX) tendency to prefer just to hope that "things might just work out" even if environment is known to be misconfigured somehow (ref: #5591).

@codecov
Copy link

codecov bot commented Apr 29, 2021

Codecov Report

Merging #5612 (9b3f0a6) into maint (26c3ccd) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##            maint    #5612      +/-   ##
==========================================
- Coverage   90.23%   90.22%   -0.02%     
==========================================
  Files         299      299              
  Lines       42234    42233       -1     
==========================================
- Hits        38109    38103       -6     
- Misses       4125     4130       +5     
Impacted Files Coverage Δ
datalad/interface/tests/test_run_procedure.py 100.00% <100.00%> (ø)
datalad/downloaders/tests/test_http.py 89.07% <0.00%> (-3.09%) ⬇️
datalad/downloaders/base.py 77.89% <0.00%> (-0.36%) ⬇️
datalad/customremotes/ria_utils.py 92.30% <0.00%> (-0.29%) ⬇️
datalad/distributed/ora_remote.py 32.37% <0.00%> (+0.35%) ⬆️
datalad/distributed/tests/test_ria_basics.py 98.36% <0.00%> (+0.40%) ⬆️
datalad/tests/utils.py 89.25% <0.00%> (+0.61%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 26c3ccd...9b3f0a6. Read the comment docs.

@yarikoptic
Copy link
Member Author

restarted one travis job, but otherwise - I think it is ready.

@yarikoptic yarikoptic added the merge-if-ok OP considers this work done, and requests review/merge label Apr 29, 2021
@kyleam kyleam merged commit 4348d36 into datalad:maint Apr 29, 2021
@yarikoptic yarikoptic deleted the bf-test branch April 29, 2021 21:55
@yarikoptic yarikoptic added semver-patch Increment the patch version when merged tests Add or improve existing tests and removed semver-patch Increment the patch version when merged labels Jun 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merge-if-ok OP considers this work done, and requests review/merge tests Add or improve existing tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants