Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix bug in aiida.engine.daemon.execmanager.retrieve_files_from_list #4275

Merged
merged 2 commits into from
Jul 22, 2020

Conversation

sphuber
Copy link
Contributor

@sphuber sphuber commented Jul 20, 2020

Fixes #4273

The retrieve_files_from_list would loop over the instructions of the
retrieve_list attribute of a calculation job and for each entry define
the variables remote_names and local_names which contain the
filenames of the remote files that are to be retrieved and with what
name locally.

However, for the code path where the element of retrieve_list is a
list or tuple and the first element contains no wildcard characters, the
remote_names variable is not defined, meaning that the value of the
previous iteration would be used. This was never detected because this
code path was not actually tested.

This bug would only affect CalcJobs that specified a retrieve_list
that contained an entry of the form:

['some/path', 'some/path', 0]

where the entry is a list and the first element does not contain a
wildcard.

@sphuber sphuber force-pushed the fix/4273/retrieve-files-from-list branch 2 times, most recently from 2e28699 to b3926f2 Compare July 20, 2020 16:43
The `retrieve_files_from_list` would loop over the instructions of the
`retrieve_list` attribute of a calculation job and for each entry define
the variables `remote_names` and `local_names` which contain the
filenames of the remote files that are to be retrieved and with what
name locally.

However, for the code path where the element of `retrieve_list` is a
list or tuple and the first element contains no wildcard characters, the
`remote_names` variable is not defined, meaning that the value of the
previous iteration would be used. This was never detected because this
code path was not actually tested.

This bug would only affect `CalcJob`s that specified a `retrieve_list`
that contained an entry of the form:

    ['some/path', 'some/path', 0]

where the entry is a list and the first element does not contain a
wildcard.
@sphuber sphuber force-pushed the fix/4273/retrieve-files-from-list branch from b3926f2 to 135caf9 Compare July 21, 2020 05:46
@codecov
Copy link

codecov bot commented Jul 21, 2020

Codecov Report

Merging #4275 into develop will increase coverage by 0.02%.
The diff coverage is 100.00%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #4275      +/-   ##
===========================================
+ Coverage    79.18%   79.19%   +0.02%     
===========================================
  Files          468      468              
  Lines        34476    34477       +1     
===========================================
+ Hits         27295    27301       +6     
+ Misses        7181     7176       -5     
Flag Coverage Δ
#django 71.12% <100.00%> (+0.02%) ⬆️
#sqlalchemy 71.95% <100.00%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
aiida/engine/daemon/execmanager.py 62.55% <100.00%> (+2.03%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0d7baa5...135caf9. Read the comment docs.

@codecov
Copy link

codecov bot commented Jul 21, 2020

Codecov Report

Merging #4275 into develop will increase coverage by 0.02%.
The diff coverage is 100.00%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #4275      +/-   ##
===========================================
+ Coverage    79.18%   79.19%   +0.02%     
===========================================
  Files          468      468              
  Lines        34475    34476       +1     
===========================================
+ Hits         27294    27300       +6     
+ Misses        7181     7176       -5     
Flag Coverage Δ
#django 71.12% <100.00%> (+0.02%) ⬆️
#sqlalchemy 71.95% <100.00%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
aiida/engine/daemon/execmanager.py 62.55% <100.00%> (+2.03%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a5829e0...4c5c04a. Read the comment docs.

@ramirezfranciscof
Copy link
Member

Since this creates the tests for this function, shouldn't we maybe add checks to make sure that you can (1) change the name of the file/folder while copying and (2) create dirs in the target directory (i.e. ['filename.txt', 'new/path/filename.txt', 0])? These seem like relevant cases that could break without noticing. Other than that I don't have many more comments, so let me know what you think of this and I'll aprove / wait for the changes and then aprove.

@sphuber
Copy link
Contributor Author

sphuber commented Jul 21, 2020

Since this creates the tests for this function, shouldn't we maybe add checks to make sure that you can (1) change the name of the file/folder while copying and (2) create dirs in the target directory (i.e. ['filename.txt', 'new/path/filename.txt', 0])? These seem like relevant cases that could break without noticing. Other than that I don't have many more comments, so let me know what you think of this and I'll aprove / wait for the changes and then aprove.

It's true that I only tested a small percentage of the pathways of this code, specifically the part that concerned the bug I am fixing there. Parts of it are tested indirectly through the integration tests that actually run some entire CalcJobs. There is no particular reason I did not add more tests other than lack of time, but I will see if I can add some more to test the other pathways as well.

@ramirezfranciscof
Copy link
Member

Yes, I agree it might not make sense to use this bugfix PR to try design the perfect coverage test for this function. I just thougth these two extra tests were particularly related to the functionality being fixed and might be worth including. Anyways, its up to you, I will aprove the PR so you can merge it if you prefer to leave these for later (or you can ping me to take a look if you decide to include them).

Copy link
Member

@ramirezfranciscof ramirezfranciscof left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to go!

@sphuber
Copy link
Contributor Author

sphuber commented Jul 22, 2020

Thanks @ramirezfranciscof . I think I will leave it as is for now. If another bug crops up, the test can be expanded

@sphuber sphuber merged commit ef1caa0 into aiidateam:develop Jul 22, 2020
@sphuber sphuber deleted the fix/4273/retrieve-files-from-list branch July 22, 2020 10:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bug in aiida.engine.daemon.execmanager.retrieve_files_from_list
2 participants