Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add python interpreter path to cfgs #5436

Merged
merged 2 commits into from
Feb 24, 2021
Merged

Conversation

mattcieslak
Copy link
Contributor

There might be an issue with the server I'm using, but I have been getting the following error when trying to create a datalad dataset with -c text2git. Using datalad 0.14.0

(base) [xcpdev@cubic-login1 production]$ datalad create -c text2git -c yoda --description "${PROJECTNAME}" . 
[INFO   ] Creating a new annex repo at /cbica/projects/xcpDev/testproj/production 
[INFO   ] Scanning for unlocked files (this may take some time) 
[INFO   ] Running procedure cfg_text2git 
[INFO   ] == Command start (output follows) ===== 
/cbica/projects/xcpDev/testproj/miniconda3/lib/python3.8/site-packages/datalad/resources/procedures/cfg_text2git.py: line 1: Procedure to configure Git annex to add text files directly to Git: command not found
import: unable to open X server `' @ error/import.c/ImportImageCommand/369.
import: unable to open X server `' @ error/import.c/ImportImageCommand/369.
/cbica/projects/xcpDev/testproj/miniconda3/lib/python3.8/site-packages/datalad/resources/procedures/cfg_text2git.py: line 6: from: command not found
/cbica/projects/xcpDev/testproj/miniconda3/lib/python3.8/site-packages/datalad/resources/procedures/cfg_text2git.py: line 8: syntax error near unexpected token `('
/cbica/projects/xcpDev/testproj/miniconda3/lib/python3.8/site-packages/datalad/resources/procedures/cfg_text2git.py: line 8: `ds = require_dataset('
[INFO   ] == Command exit (modification check follows) ===== 
CommandError: '/cbica/projects/xcpDev/testproj/miniconda3/lib/python3.8/site-packages/datalad/resources/procedures/cfg_text2git.py /cbica/projects/xcpDev/testproj/production ' failed with exitcode 2 under /cbica/projects/xcpDev/testproj/production

This fixes the issue for me, but please let me know if there is something I can configure so this isn't necessary.

@codecov
Copy link

codecov bot commented Feb 24, 2021

Codecov Report

Merging #5436 (aa66c7d) into master (bc8e364) will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #5436   +/-   ##
=======================================
  Coverage   89.47%   89.47%           
=======================================
  Files         296      296           
  Lines       41794    41794           
=======================================
  Hits        37396    37396           
  Misses       4398     4398           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bc8e364...a1ea8b2. Read the comment docs.

@yarikoptic
Copy link
Member

Are those procedures files executable? Any idea why/how?

Here is our logic to decide to either execute or interpret
https://github.com/datalad/datalad/blob/master/datalad/interface/run_procedure.py#L162

Having said that I don't mind at all adding the shebangs - costs us nothing but could indeed mitigate such cases . I only wonder if it might better be python3 and as python might not even be there (Debian eg is getting rid of pure python use to iirc ensure complete migration over to python3)

@bpoldrack
Copy link
Member

bpoldrack commented Feb 24, 2021

Don't mind it, too. But I think this is indeed the issue to figure out, since it's likely true for other procedures as well:

Are those procedures files executable? Any idea why/how?

@mattcieslak
Copy link
Contributor Author

the datalad installation was through pip, which was installed in through conda on a CentOS 7 x86_64 machine. I checked how this file would run through your function:

>>> import os, stat
>>> script_file = "/cbica/projects/xcpDev/testproj/miniconda3/lib/python3.8/site-packages/datalad/resources/procedures/cfg_text2git.py"
>>> os.stat(script_file).st_mode
33256
>>> os.stat(script_file).st_mode & stat.S_IEXEC
64

And to double-check:

>>> from datalad.interface.run_procedure import _guess_exec
>>> _guess_exec(script_file) 
{'type': 'executable',
 'template': '{script} {ds} {args}',
 'state': 'executable'}

So it does look like these are executable files.

@yarikoptic
Copy link
Member

I am still curious -- what is the output of

ls -l /cbica/projects/xcpDev/testproj/miniconda3/lib/python3.8/site-packages/datalad /cbica/projects/xcpDev/testproj/miniconda3/lib/python3.8/site-packages/datalad/resources/procedures/

and

mount | grep /cbica

@mattcieslak ? I still wonder if that is may be some "interesting" filesystem used on the beast rendering all files executable.

just for a check, I have downloaded the datalad-0.14.0.tar.gz from pypi and it seems to have only expected scripts to be executable, so it still remains a mystery somewhat
lena:/tmp
$> tar -tzvf datalad-0.14.0.tar.gz | grep 'x kyle' | grep -v '/$'
-rwxr-xr-x kyle/kyle       223 2020-03-16 12:40 datalad-0.14.0/benchmarks/scripts/heavyout
-rwxr-xr-x kyle/kyle      6423 2021-01-26 14:49 datalad-0.14.0/setup.py
-rwxr-xr-x kyle/kyle      1283 2020-03-16 12:40 datalad-0.14.0/tools/bisect-git-annex
-rwxr-xr-x kyle/kyle       241 2020-03-16 12:40 datalad-0.14.0/tools/bisect-git-annex.scripts/bisect-git-annex-doublepasswd.sh
-rwxr-xr-x kyle/kyle       197 2020-03-16 12:40 datalad-0.14.0/tools/bisect-git-annex.scripts/bisect-git-annex-lock.sh
-rwxr-xr-x kyle/kyle      4111 2020-05-22 17:30 datalad-0.14.0/tools/cast2asciinema
-rwxr-xr-x kyle/kyle      2117 2020-03-16 12:40 datalad-0.14.0/tools/cast2narration
-rwxr-xr-x kyle/kyle      1672 2020-03-16 12:40 datalad-0.14.0/tools/cast2rst
-rwxr-xr-x kyle/kyle       656 2020-03-16 12:40 datalad-0.14.0/tools/cast2script
-rwxr-xr-x kyle/kyle      3064 2020-03-16 12:40 datalad-0.14.0/tools/cast_live
-rwxr-xr-x kyle/kyle       884 2020-05-22 17:30 datalad-0.14.0/tools/changelog-todo
-rwxr-xr-x kyle/kyle      2177 2020-05-22 17:30 datalad-0.14.0/tools/checkpwd.py
-rwxr-xr-x kyle/kyle       120 2021-01-26 14:49 datalad-0.14.0/tools/ci/appveyor_ssh2localhost.sh
-rwxr-xr-x kyle/kyle       812 2020-03-16 12:40 datalad-0.14.0/tools/ci/benchmark-travis-pr.sh
-rwxr-xr-x kyle/kyle      2943 2021-01-26 14:49 datalad-0.14.0/tools/ci/bisect-python.sh
-rwxr-xr-x kyle/kyle      2481 2021-02-02 12:32 datalad-0.14.0/tools/ci/download-latest-artifact
-rwxr-xr-x kyle/kyle      8845 2020-10-06 13:45 datalad-0.14.0/tools/ci/install-annex.sh
-rwxr-xr-x kyle/kyle       377 2020-10-06 13:45 datalad-0.14.0/tools/ci/install-minimum-git.sh
-rwxr-xr-x kyle/kyle       218 2020-08-10 15:38 datalad-0.14.0/tools/ci/install-upstream-git.sh
-rwxr-xr-x kyle/kyle       715 2020-03-16 12:40 datalad-0.14.0/tools/ci/prep-travis-devel-annex.sh
-rwxr-xr-x kyle/kyle       886 2021-01-26 14:49 datalad-0.14.0/tools/ci/prep-travis-forssh.sh
-rwxr-xr-x kyle/kyle      2351 2021-01-26 14:49 datalad-0.14.0/tools/copy_urls_from_datalad.py
-rwxr-xr-x kyle/kyle       761 2020-10-30 16:03 datalad-0.14.0/tools/coverage-bin/datalad
-rwxr-xr-x kyle/kyle       761 2020-10-30 16:03 datalad-0.14.0/tools/coverage-bin/git-annex-remote-datalad
-rwxr-xr-x kyle/kyle       761 2020-10-30 16:03 datalad-0.14.0/tools/coverage-bin/git-annex-remote-datalad-archives
-rwxr-xr-x kyle/kyle      1007 2020-05-22 17:30 datalad-0.14.0/tools/describegitannex
-rwxr-xr-x kyle/kyle      1791 2020-10-30 16:03 datalad-0.14.0/tools/downgrade-annex
-rwxr-xr-x kyle/kyle      1531 2020-03-16 12:40 datalad-0.14.0/tools/dtime
-rwxr-xr-x kyle/kyle       806 2020-08-28 10:54 datalad-0.14.0/tools/eval_under_testloopfs
-rwxr-xr-x kyle/kyle      2127 2020-03-16 12:40 datalad-0.14.0/tools/git-web-submodules.sh
-rwxr-xr-x kyle/kyle       447 2020-03-16 12:40 datalad-0.14.0/tools/link_issues_CHANGELOG
-rwxr-xr-x kyle/kyle      3010 2020-03-16 12:40 datalad-0.14.0/tools/mimic_merges
-rwxr-xr-x kyle/kyle      4562 2020-03-16 12:40 datalad-0.14.0/tools/mimic_repo
-rwxr-xr-x kyle/kyle       419 2020-03-16 12:40 datalad-0.14.0/tools/mkcontrib
-rwxr-xr-x kyle/kyle      2797 2020-03-16 12:40 datalad-0.14.0/tools/monitor-interrupts.py
-rwxr-xr-x kyle/kyle      2298 2020-03-16 12:40 datalad-0.14.0/tools/profile_python
-rwxr-xr-x kyle/kyle      3457 2020-03-16 12:40 datalad-0.14.0/tools/testing/make_test_repo
-rwxr-xr-x kyle/kyle       938 2020-03-16 12:40 datalad-0.14.0/tools/testing/make_test_repos
-rwxr-xr-x kyle/kyle      4134 2020-03-16 12:40 datalad-0.14.0/tools/testing/start_website_in_docker
-rwxr-xr-x kyle/kyle      1932 2020-05-22 17:30 datalad-0.14.0/tools/testing/test_README_in_docker
-rwxr-xr-x kyle/kyle       481 2020-03-16 12:40 datalad-0.14.0/tools/testing/travis_ifdown_nonlo.sh
-rwxr-xr-x kyle/kyle       593 2020-03-16 12:40 datalad-0.14.0/tools/upgrade-annex-osx.sh
-rwxr-xr-x kyle/kyle      3565 2020-03-16 12:40 datalad-0.14.0/tools/urlinfo

@mattcieslak
Copy link
Contributor Author

Here are the outputs:

$ ls -l /cbica/projects/xcpDev/testproj/miniconda3/lib/python3.8/site-packages/datalad /cbica/projects/xcpDev/testproj/miniconda3/lib/python3.8/site-packages/datalad/resources/procedures/
/cbica/projects/xcpDev/testproj/miniconda3/lib/python3.8/site-packages/datalad:
total 906
-rwxr-x---. 1 xcpdev xcpdev  3562 Feb 23 20:46 api.py*
-rwxr-x---. 1 xcpdev xcpdev 13060 Feb 23 20:46 auto.py*
drwxr-x---. 4 xcpdev xcpdev  4096 Feb 23 20:46 cmdline/
-rwxr-x---. 1 xcpdev xcpdev 32659 Feb 23 20:46 cmd.py*
-rwxr-x---. 1 xcpdev xcpdev 37155 Feb 23 20:46 config.py*
-rwxr-x---. 1 xcpdev xcpdev  2937 Feb 23 20:46 consts.py*
drwxr-x---. 5 xcpdev xcpdev  4096 Feb 23 20:46 core/
-rwxr-x---. 1 xcpdev xcpdev  1333 Feb 23 20:46 coreapi.py*
drwxr-x---. 4 xcpdev xcpdev  4096 Feb 23 20:46 customremotes/
drwxr-x---. 4 xcpdev xcpdev  4096 Feb 23 20:46 distributed/
drwxr-x---. 4 xcpdev xcpdev  4096 Feb 23 20:46 distribution/
-rwxr-x---. 1 xcpdev xcpdev 11479 Feb 23 20:46 dochelpers.py*
drwxr-x---. 5 xcpdev xcpdev  4096 Feb 23 20:46 downloaders/
-rwxr-x---. 1 xcpdev xcpdev  9234 Feb 23 20:46 __init__.py*
drwxr-x---. 4 xcpdev xcpdev  4096 Feb 23 20:46 interface/
drwxr-x---. 4 xcpdev xcpdev  4096 Feb 23 20:46 local/
-rwxr-x---. 1 xcpdev xcpdev 20708 Feb 23 20:46 log.py*
-rwxr-x---. 1 xcpdev xcpdev  3132 Feb 23 20:46 __main__.py*
drwxr-x---. 6 xcpdev xcpdev  4096 Feb 23 20:46 metadata/
drwxr-x---. 4 xcpdev xcpdev  4096 Feb 23 20:46 plugin/
drwxr-x---. 2 xcpdev xcpdev  4096 Feb 23 20:46 __pycache__/
drwxr-x---. 4 xcpdev xcpdev  4096 Feb 23 20:46 resources/
drwxr-x---. 5 xcpdev xcpdev  4096 Feb 23 20:46 support/
drwxr-x---. 3 xcpdev xcpdev  4096 Feb 23 20:46 tests/
drwxr-x---. 4 xcpdev xcpdev  4096 Feb 23 20:46 ui/
-rwxr-x---. 1 xcpdev xcpdev 85653 Feb 23 20:46 utils.py*
-rwxr-x---. 1 xcpdev xcpdev  2179 Feb 23 20:46 version.py*

/cbica/projects/xcpDev/testproj/miniconda3/lib/python3.8/site-packages/datalad/resources/procedures/:
total 2
-rwxr-x---. 1 xcpdev xcpdev  820 Feb 23 20:46 cfg_metadatatypes.py*
-rwxr-x---. 1 xcpdev xcpdev  706 Feb 23 20:57 cfg_text2git.py*
-rwxr-x---. 1 xcpdev xcpdev 2225 Feb 23 20:57 cfg_yoda.py*
drwxr-x---. 2 xcpdev xcpdev 4096 Feb 23 20:46 __pycache__/

mount | grep /cbica produces a ton of output and I'll have to check with the sysadmin if I can post it here. It's some kind of convoluted gpfs.

@yarikoptic
Copy link
Member

No need to pester admin -- so it is gpfs... for some reason it announces all files to be "executable" as a good old FAT ;) uff... so we definitely should provide shebangs . Would you mind

  • enhancing your PR with shebangs for all other procedures?
  • rebasing it on top of the maint branch and making this PR's base to be maint, not master?
    if the last one is a bit too much -- that is ok, I will just then rebase "manually" locally for the merge.

@yarikoptic
Copy link
Member

hold on -- we do not have more procedures in the datalad core... so let's consider that done ;) let me know about rebase

@mattcieslak
Copy link
Contributor Author

I may have screwed up the rebase

@kyleam
Copy link
Contributor

kyleam commented Feb 24, 2021

I may have screwed up the rebase

No worries. With your PR's branch checked out, git rebase --onto maint master would move just the PR's commits on top of maint.

I'll push a fixup.

@kyleam kyleam changed the base branch from master to maint February 24, 2021 18:00
@mattcieslak
Copy link
Contributor Author

Thanks!

@kyleam
Copy link
Contributor

kyleam commented Feb 24, 2021

The only test failures are the macos startup failures we've recently started seeing. Merging.

Thanks @mattcieslak

@kyleam kyleam merged commit 8eaf2cf into datalad:maint Feb 24, 2021
kyleam added a commit that referenced this pull request Feb 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants