Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

datalad 0.14.0rc1 breaks test_orc_datalad_pair_new_submodule #568

Closed
yarikoptic opened this issue Jan 28, 2021 · 0 comments · Fixed by #569
Closed

datalad 0.14.0rc1 breaks test_orc_datalad_pair_new_submodule #568

yarikoptic opened this issue Jan 28, 2021 · 0 comments · Fixed by #569
Assignees
Labels

Comments

@yarikoptic
Copy link
Member

discovered while testing #567

copy/paste of output for test_orc_datalad_pair_new_submodule from raw log
�[32mINFO    �[0m datalad.gitrepo:log.py:323 Finished writing objects
�[31m�[1m_____________________ test_orc_datalad_pair_new_submodule ______________________�[0m

job_spec = {'_command_array': ["sh -c 'echo a >sub/a'"], '_extra_inputs_array': [[]], '_inputs_array': [[]], '_outputs_array': [[]], ...}
dataset = Dataset('/tmp/pytest-of-travis/pytest-0/dataset33')
shell = Shell(name='localshell', id=None, type='shell', status='available')

    �[94mdef�[39;49;00m �[92mtest_orc_datalad_pair_new_submodule�[39;49;00m(job_spec, dataset, shell):
        �[94mwith�[39;49;00m chpwd(dataset.path):
            orc = orcs.DataladPairOrchestrator(
                shell, submission_type=�[33m"�[39;49;00m�[33mlocal�[39;49;00m�[33m"�[39;49;00m, job_spec=job_spec)
            orc.prepare_remote()
            orc.submit()
            orc.follow()
            orc.fetch()
    
            �[90m# prepare_remote() doesn't fail when a new subdataset is added after�[39;49;00m
            �[90m# the first run.�[39;49;00m
            sub = dataset.create(�[33m"�[39;49;00m�[33msub�[39;49;00m�[33m"�[39;49;00m)
            dataset.save()
    
            job_spec[�[33m"�[39;49;00m�[33m_resolved_command_str�[39;49;00m�[33m"�[39;49;00m] = �[33m"�[39;49;00m�[33msh -c �[39;49;00m�[33m'�[39;49;00m�[33mecho a >sub/a�[39;49;00m�[33m'�[39;49;00m�[33m"�[39;49;00m
            job_spec[�[33m"�[39;49;00m�[33minputs�[39;49;00m�[33m"�[39;49;00m] = []
            job_spec[�[33m"�[39;49;00m�[33moutputs�[39;49;00m�[33m"�[39;49;00m] = []
    
            orc = orcs.DataladPairOrchestrator(
                shell, submission_type=�[33m"�[39;49;00m�[33mlocal�[39;49;00m�[33m"�[39;49;00m, job_spec=job_spec)
>           orc.prepare_remote()

�[1m�[31mreproman/support/jobs/tests/test_orchestrators.py�[0m:867: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
�[1m�[31mreproman/support/jobs/orchestrators.py�[0m:830: in prepare_remote
    �[96mself�[39;49;00m._fix_up_dataset()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <reproman.support.jobs.orchestrators.DataladPairOrchestrator object at 0x7f2d847dd7f0>

    �[94mdef�[39;49;00m �[92m_fix_up_dataset�[39;49;00m(�[96mself�[39;49;00m):
        �[33m"""Try to get datataset and subdatasets into the correct state.�[39;49;00m
    �[33m    """�[39;49;00m
        �[96mself�[39;49;00m._checkout_target()
        �[90m# fixup 0: 'datalad create-sibling --recursive' leaves the subdataset�[39;49;00m
        �[90m# uninitialized (see DataLad's 78e00dcd2).�[39;49;00m
        �[96mself�[39;49;00m._execute_in_wdir([�[33m"�[39;49;00m�[33mgit�[39;49;00m�[33m"�[39;49;00m, �[33m"�[39;49;00m�[33msubmodule�[39;49;00m�[33m"�[39;49;00m, �[33m"�[39;49;00m�[33mupdate�[39;49;00m�[33m"�[39;49;00m, �[33m"�[39;49;00m�[33m--init�[39;49;00m�[33m"�[39;49;00m])
    
        �[90m# fixup 1: Check out target commit in subdatasets. This should later be�[39;49;00m
        �[90m# replaced by the planned Datalad functionality to get an entire�[39;49;00m
        �[90m# dataset hierarchy to a recorded state.�[39;49;00m
        �[90m#�[39;49;00m
        �[90m# fixup 2: Autoenable remotes:�[39;49;00m
        �[90m# 'datalad publish' does not autoenable remotes, and 'datalad�[39;49;00m
        �[90m# create-sibling' calls 'git annex init' too early to trigger�[39;49;00m
        �[90m# autoenabling. Temporarily work around this issue, though this�[39;49;00m
        �[90m# should very likely be addressed in DataLad. And if this is here�[39;49;00m
        �[90m# to stay, we should avoid this call for non-annex datasets.�[39;49;00m
        lgr.info(�[33m"�[39;49;00m�[33mAdjusting state of remote dataset�[39;49;00m�[33m"�[39;49;00m)
        �[96mself�[39;49;00m._execute_in_wdir([�[33m"�[39;49;00m�[33mgit�[39;49;00m�[33m"�[39;49;00m, �[33m"�[39;49;00m�[33mannex�[39;49;00m�[33m"�[39;49;00m, �[33m"�[39;49;00m�[33minit�[39;49;00m�[33m"�[39;49;00m])
        �[94mfor�[39;49;00m res �[95min�[39;49;00m �[96mself�[39;49;00m._execute_datalad_json_command(
                [�[33m"�[39;49;00m�[33msubdatasets�[39;49;00m�[33m"�[39;49;00m, �[33m"�[39;49;00m�[33m--fulfilled=true�[39;49;00m�[33m"�[39;49;00m, �[33m"�[39;49;00m�[33m--recursive�[39;49;00m�[33m"�[39;49;00m]):
            cwd = res[�[33m"�[39;49;00m�[33mpath�[39;49;00m�[33m"�[39;49;00m]
            �[96mself�[39;49;00m._assert_clean_repo(cwd=cwd)
            lgr.debug(�[33m"�[39;49;00m�[33mAdjusting state of �[39;49;00m�[33m%s�[39;49;00m�[33m"�[39;49;00m, cwd)
>           cmds = [[�[33m"�[39;49;00m�[33mgit�[39;49;00m�[33m"�[39;49;00m, �[33m"�[39;49;00m�[33mcheckout�[39;49;00m�[33m"�[39;49;00m, res[�[33m"�[39;49;00m�[33mrevision�[39;49;00m�[33m"�[39;49;00m]],
                    [�[33m"�[39;49;00m�[33mgit�[39;49;00m�[33m"�[39;49;00m, �[33m"�[39;49;00m�[33mannex�[39;49;00m�[33m"�[39;49;00m, �[33m"�[39;49;00m�[33minit�[39;49;00m�[33m"�[39;49;00m]]
�[1m�[31mE           KeyError: 'revision'�[0m

�[1m�[31mreproman/support/jobs/orchestrators.py�[0m:754: KeyError

we installed rc:

Collecting datalad>=0.13.0rc2
  Downloading datalad-0.14.0rc1.tar.gz (1.6 MB)

It also failed similarly in recent master run but was all green a few days back (whenever there were no datalad rc) https://travis-ci.org/github/ReproNim/reproman/builds/755466822 . So -- must be datalad rc related .

kyleam added a commit to kyleam/niceman that referenced this issue Jan 29, 2021
In DataLad v0.12.0 (c89746c27e), the "revision" key in the results for
`datalad subdatasets` was changed from "revision" to "gitshasum", and
the old name was kept around for compatibility.  The old key will be
removed in the upcoming 0.14 release (f5c3104d88).

Switch to using "gitshasum".  (Our minimum DataLad version is 0.13, so
a compatibility kludge isn't needed.)

Fixes ReproNim#568.
@kyleam kyleam added the fixed label Jan 29, 2021
kyleam added a commit to kyleam/niceman that referenced this issue Jan 29, 2021
In DataLad v0.12.0 (c89746c27e), the "revision" key in the results for
`datalad subdatasets` was changed from "revision" to "gitshasum", and
the old name was kept around for compatibility.  The old key will be
removed in the upcoming 0.14 release (f5c3104d88).

Look for "gitshasum", but fall back to "revision" because the DataLad
version on the remote may be below the version specified in setup.py
(currently 0.13).

Fixes ReproNim#568.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants