[BUG]: No error is logged if datalad fails to clone a repository #235

synchon · 2023-06-28T14:04:50Z

Is there an existing issue for this?

I have searched the existing issues

Current Behavior

This is the standard error:

[INFO] Attempting a clone into /tmp/tmpmms6ilm3/datadir 
[INFO] Attempting to clone from https://github.com/datalad-datasets/human-connectome-project-openaccess.git to /tmp/tmpmms6ilm3/datadir 
[INFO] Attempting to clone from https://github.com/datalad-datasets/human-connectome-project-openaccess.git/.git to /tmp/tmpmms6ilm3/datadir 
[INFO] Completed clone attempts for Dataset(/tmp/tmpmms6ilm3/datadir) 
Traceback (most recent call last):
  File "/home/fraimondo/miniconda3/envs/afni/bin/junifer", line 8, in <module>
    sys.exit(cli())
  File "/home/fraimondo/miniconda3/envs/afni/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/fraimondo/miniconda3/envs/afni/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/fraimondo/miniconda3/envs/afni/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/fraimondo/miniconda3/envs/afni/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/fraimondo/miniconda3/envs/afni/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/fraimondo/dev/tbox/junifer/junifer/api/cli.py", line 160, in run
    api_run(
  File "/home/fraimondo/dev/tbox/junifer/junifer/api/functions.py", line 164, in run
    with datagrabber_object:
  File "/home/fraimondo/dev/tbox/junifer/junifer/datagrabber/multiple.py", line 81, in __enter__
    dg.__enter__()
  File "/home/fraimondo/dev/tbox/junifer/junifer/datagrabber/datalad_base.py", line 293, in __enter__
    self.install()
  File "/home/fraimondo/dev/tbox/junifer/junifer/datagrabber/datalad_base.py", line 244, in install
    self._dataset: dl.Dataset = dl.clone(  # type: ignore
  File "/home/fraimondo/dev/tbox/datalad/datalad/interface/base.py", line 773, in eval_func
    return return_func(*args, **kwargs)
  File "/home/fraimondo/dev/tbox/datalad/datalad/interface/base.py", line 763, in return_func
    results = list(results)
  File "/home/fraimondo/dev/tbox/datalad/datalad/interface/base.py", line 940, in _execute_command_
    raise IncompleteResultsError(
datalad.support.exceptions.IncompleteResultsError: Command did not complete successfully. 1 failed:
[{'action': 'install',
  'message': ('Failed to clone from any candidate source URL. Encountered '
              'errors per each url were:\n'
              '- %s',
              'https://github.com/datalad-datasets/human-connectome-project-openaccess.git\n'
              "  CommandError: 'git -c diff.ignoreSubmodules=none clone "
              '--progress '
              'https://github.com/datalad-datasets/human-connectome-project-openaccess.git '
              "/tmp/tmpmms6ilm3/datadir' failed with exitcode 128 [err: "
              "'Cloning into '/tmp/tmpmms6ilm3/datadir'...\n"
              'fatal: unable to access '
              "'https://github.com/datalad-datasets/human-connectome-project-openaccess.git/': "
              "The requested URL returned error: 403']\n"
              '- '
              'https://github.com/datalad-datasets/human-connectome-project-openaccess.git/.git\n'
              "  CommandError: 'git -c diff.ignoreSubmodules=none clone "
              '--progress '
              'https://github.com/datalad-datasets/human-connectome-project-openaccess.git/.git '
              "/tmp/tmpmms6ilm3/datadir' failed with exitcode 128 [err: "
              "'Cloning into '/tmp/tmpmms6ilm3/datadir'...\n"
              'fatal: unable to access '
              "'https://github.com/datalad-datasets/human-connectome-project-openaccess.git/.git/': "
              "The requested URL returned error: 403']"),
  'path': '/tmp/tmpmms6ilm3/datadir',
  'source': {'default_destpath': 'human-connectome-project-openaccess',
             'giturl': 'https://github.com/datalad-datasets/human-connectome-project-openaccess.git/.git',
             'source': 'https://github.com/datalad-datasets/human-connectome-project-openaccess.git',
             'type': 'giturl',
             'version': None},
  'source_url': 'https://github.com/datalad-datasets/human-connectome-project-openaccess.git',
  'status': 'error',
  'type': 'dataset'}]

This is the standard output:

2023-06-28 09:13:07,958 - JUNIFER - INFO - ===== Lib Versions =====
2023-06-28 09:13:07,958 - JUNIFER - INFO - numpy: 1.23.5
2023-06-28 09:13:07,958 - JUNIFER - INFO - scipy: 1.11.0
2023-06-28 09:13:07,958 - JUNIFER - INFO - pandas: 1.5.3
2023-06-28 09:13:07,958 - JUNIFER - INFO - nilearn: 0.10.0
2023-06-28 09:13:07,958 - JUNIFER - INFO - nibabel: 4.0.2
2023-06-28 09:13:07,958 - JUNIFER - INFO - junifer: 0.0.3.dev101
2023-06-28 09:13:07,958 - JUNIFER - INFO - ========================
2023-06-28 09:13:07,958 - JUNIFER - INFO - Parsing yaml file: /home/fraimondo/dev/projects/brain_size/1_compute_features/junifer_jobs/HCP_icbm152_mask_5mmreho/config.yaml
2023-06-28 09:13:07,970 - JUNIFER - INFO - Registering HCPCATConfounds in datagrabber
2023-06-28 09:13:07,970 - JUNIFER - INFO - Registering MultipleHCP in datagrabber
2023-06-28 09:13:07,970 - JUNIFER - INFO - `datadir` is None, creating a temporary directory
2023-06-28 09:13:07,971 - JUNIFER - INFO - `datadir` set to /tmp/tmpmms6ilm3/datadir
2023-06-28 09:13:07,972 - JUNIFER - INFO - `datadir` is None, creating a temporary directory
2023-06-28 09:13:07,972 - JUNIFER - INFO - `datadir` set to /tmp/tmp1bzrmjo4/datadir

Expected Behavior

an error being logged in the standard output

Steps To Reproduce

This is hard to tell, maybe the best way to reproduce this is to try to run a test using a datalad dataset while not connected to the internet.

Environment

junifer:
  version: 0.0.3.dev101
python:
  version: 3.10.12
  implementation: CPython
dependencies:
  click: 8.1.3
  numpy: 1.23.5
  datalad: 0.18.2+59.gc5054cb91
  pandas: 1.5.3
  nibabel: 4.0.2
  nilearn: 0.10.0
  sqlalchemy: 1.4.46
  ruamel.yaml: 0.17.32
system:
  platform: Linux-6.1.0-9-amd64-x86_64-with-glibc2.36
environment:
  LC_CTYPE: en_US.UTF-8
  PATH: 
    /home/fraimondo/miniconda3/envs/afni/bin:/home/fraimondo/miniconda3/condabin:/home/fraimondo/.dotfiles/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin:/usr/local/games:/usr/games

Relevant log output

No response

Anything else?

No response

github-actions · 2023-06-28T14:12:01Z

PR Preview Action v1.4.4
🚀 Deployed preview to https://juaml.github.io/junifer/pr-preview/pr-235/
on branch `gh-pages` at 2023-06-28 14:19 UTC

codecov · 2023-06-28T14:14:47Z

Codecov Report

Merging #235 (9d30d1f) into main (c811a65) will decrease coverage by 0.06%.
The diff coverage is 50.00%.

@@            Coverage Diff             @@
##             main     #235      +/-   ##
==========================================
- Coverage   93.61%   93.56%   -0.06%     
==========================================
  Files          82       82              
  Lines        3460     3463       +3     
  Branches      644      644              
==========================================
+ Hits         3239     3240       +1     
- Misses        144      146       +2     
  Partials       77       77

Flag	Coverage Δ
docs	`100.00% <ø> (ø)`
junifer	`93.55% <50.00%> (-0.06%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
junifer/datagrabber/datalad_base.py	`89.23% <50.00%> (-1.33%)`	⬇️

... and 2 files with indirect coverage changes

fraimondo · 2023-06-28T14:52:05Z

@synchon What about the get operations. Are we logging errors there?

synchon · 2023-06-28T15:13:19Z

@synchon What about the get operations. Are we logging errors there?

Yes we are:

junifer/junifer/datagrabber/datalad_base.py

Lines 188 to 191 in c458035

    
           try: 
        
               dl_out = self._dataset.get(to_get, result_renderer="disabled") 
        
           except IncompleteResultsError as e: 
        
               raise_error(f"Failed to get from dataset: {e.failed}")

fraimondo added bug Something isn't working triage New issues waiting to be reviewed labels Jun 28, 2023

fraimondo assigned synchon Jun 28, 2023

fraimondo removed the triage New issues waiting to be reviewed label Jun 28, 2023

fix: handle datalad exception for partial clone in DataladDataGrabber

4a25560

chore: add changelog 235.bugfix

9d30d1f

synchon requested a review from fraimondo June 28, 2023 14:12

fraimondo approved these changes Jun 28, 2023

View reviewed changes

synchon merged commit 4233265 into main Jun 29, 2023

synchon deleted the fix/datalad-get-fail-err branch June 29, 2023 05:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG]: No error is logged if datalad fails to clone a repository #235

[BUG]: No error is logged if datalad fails to clone a repository #235

synchon commented Jun 28, 2023

github-actions bot commented Jun 28, 2023 •

edited

Loading

codecov bot commented Jun 28, 2023 •

edited

Loading

fraimondo commented Jun 28, 2023

synchon commented Jun 28, 2023

[BUG]: No error is logged if datalad fails to clone a repository #235

[BUG]: No error is logged if datalad fails to clone a repository #235

Conversation

synchon commented Jun 28, 2023

Is there an existing issue for this?

Current Behavior

Expected Behavior

Steps To Reproduce

Environment

Relevant log output

Anything else?

github-actions bot commented Jun 28, 2023 • edited Loading

codecov bot commented Jun 28, 2023 • edited Loading

Codecov Report

fraimondo commented Jun 28, 2023

synchon commented Jun 28, 2023

github-actions bot commented Jun 28, 2023 •

edited

Loading

codecov bot commented Jun 28, 2023 •

edited

Loading