✨ NEW: Add `verdi group move-nodes` command #4428

mbercx · 2020-10-07T19:33:24Z

Add an extra subcommand move-nodes to verdi group that moves nodes
from a source group to a target group, instead of having to use the
remove-nodes and add-nodes subcommands in order to achieve this.

Similar to the remove-nodes and add-nodes commands, the move can be
made without prompts by adding the --force flag, and the nodes are
presented as a list of arguments.

Add corresponding test method to the TestVerdiGroup class of the
test_group.py suite.

ramirezfranciscof

It all looks pretty good! I just added two ideas that might be worth considering, let me know what you think.

aiida/cmdline/commands/cmd_group.py

tests/cmdline/commands/test_group.py

mbercx · 2020-10-09T15:54:20Z

Thanks for the review, @ramirezfranciscof! I've replied to your comments, will make changes once we have resolved the issues.

mbercx · 2021-02-16T19:20:21Z

Usage:

$ verdi group move-nodes -h
Usage: verdi group move-nodes [OPTIONS] [NODES]...

  Move nodes from one group to another.

Options:
  -s, --source-group GROUP  A single group identified by its ID, UUID or
                            label.  [required]

  -t, --target-group GROUP  A single group identified by its ID, UUID or
                            label.  [required]

  -f, --force               Do not ask for confirmation and skip all checks.
  -h, --help                Show this message and exit.

I've created two test groups with the following content:

$ verdi group show test1
-----------------  ----------------
Group label        test1
Group type_string  core
Group description  <no description>
-----------------  ----------------
# Nodes:
  PK  Type    Created
----  ------  --------------
 161  Int     7D:10h:42m ago
 157  Int     7D:10h:44m ago
 159  Int     7D:10h:43m ago

$ verdi group show test2
-----------------  ----------------
Group label        test2
Group type_string  core
Group description  <no description>
-----------------  ----------------
# Nodes:
  PK  Type    Created
----  ------  --------------
 159  Int     7D:10h:43m ago

When I try to move a node that is not in test1 to test2:

$ verdi group move-nodes -s test1 -t test2 151
Critical: None of the specified nodes are in Group<test1>.

When I try to move two nodes, one which is in test1 and one that isn't:

$ verdi group move-nodes -s test1 -t test2 151 161
Warning: 1 nodes with PK {151} are not in Group<test1>.
Are you sure you want to move 1 nodes from Group<test1> to Group<test2>? [y/N]:

When I try to move three nodes, two which is in test1 and one that isn't, but one of the two that is in test1 is also in test2:

$ verdi group move-nodes -s test1 -t test2 161 151 159
Warning: 1 nodes with PK {151} are not in Group<test1>.
Warning: 1 nodes with PK {159} are already in Group<test2>. These will still be removed from Group<test1>.
Are you sure you want to move 2 nodes from Group<test1> to Group<test2>? [y/N]:

When I try to move no nodes at all:

$ verdi group move-nodes -s test1 -t test2
Critical: No nodes identifiers were provided.

When I try to move the nodes to the same target group as the source group:

$ verdi group move-nodes -s test1 -t test1 161
Critical: Source and target group are the same: Group<test1>

When I come to my senses, and try to move two nodes which are in test1, but aren't in test2:

$ verdi group move-nodes -s test1 -t test2 161 157
Are you sure you want to move 2 nodes from Group<test1> to Group<test2>? [y/N]:

mbercx · 2021-02-16T19:36:32Z

@ramirezfranciscof I've updated the PR in line with with the changes that were made for the verdi group remove-nodes command in cba7d6d. I've added examples of the usage above, and described the warnings/failures added in the commit message of 7bef6a3. I will update the tests and add documentation when we're agreed on how it should work.

To come back to your original worry about the source_group.remove_nodes(nodes) failing: I think this is either not an issue or at least not an issue that should be solved here. The verdi group remove-nodes also calls this command. What if it fails there? In case the command fails here, the issue is the same. However, just glancing at the remove_nodes method for the Group class and its backends, I think it's written pretty robustly, so that it will only remove any of the nodes if there are no issues.

I've also added @yakutovicha as a reviewer, since he actually requested to be a reviewer for the move-nodes command, not the changes I made for the remove-nodes command. 😅

codecov · 2021-02-16T19:36:40Z

Codecov Report

Merging #4428 (d1d98c9) into develop (7017f30) will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           develop    #4428      +/-   ##
===========================================
+ Coverage    82.11%   82.13%   +0.02%     
===========================================
  Files          533      533              
  Lines        38432    38470      +38     
===========================================
+ Hits         31554    31592      +38     
  Misses        6878     6878

Flag	Coverage Δ
django	`77.21% <100.00%> (+0.03%)`	⬆️
sqlalchemy	`76.51% <100.00%> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
aiida/cmdline/commands/cmd_group.py	`91.62% <100.00%> (+1.23%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7017f30...d1d98c9. Read the comment docs.

yakutovicha

Thanks, @mbercx, I think you did a great job. I also agree with the logic you've come up with to make the moving process as safe as possible: first, decide what to do and only then do it.

I have to admit though, that unrelated to this PR we have to add some functionality to the groups to simplify the code and to make things faster:

In principle, we don't need to load nodes when adding them to the group, but we can't use the pk or UUID to reference the node. I made an issue some years ago about that (Group.add_nodes() function should be able to get a list of 'pk' or 'uuid' as the argument. #1320).
We should be able to directly get the pks of the nodes that belong to a group. I made an issue some seconds ago about that (Implement group.node_pks #4755)

aiida/cmdline/commands/cmd_group.py

mbercx · 2021-02-17T10:42:28Z

Thanks for the review @yakutovicha! I agree with both your points, and would still add that perhaps there should be a click option that just passes the identifiers without loading the nodes first, removing the need for e.g. this line:

node_pks = [node.pk for node in nodes]

Then we can execute these group commands and their checks without ever having to load the nodes. However, somewhere there should be a check that the identifiers do indeed correspond to nodes.

If it's ok for you, I'll keep the slow version for now, and open an issue regarding improving the speed of these commands that refers to your issues, which we will solve for v2.0.0.

yakutovicha · 2021-02-17T10:45:21Z

If it's ok for you, I'll keep the slow version for now, and open an issue regarding improving the speed of these commands that refers to your issues, which we will solve for v2.0.0.

Definitely. From my side, the PR is good to be merged. Should I approve or do we want @ramirezfranciscof to have a look as well?

yakutovicha · 2021-02-17T10:45:44Z

provided that the pre-commit issue is fixed, of course :)

mbercx · 2021-02-17T11:00:58Z

provided that the pre-commit issue is fixed, of course :)

For the record, that pre-commit issue is not from files changed in this PR. :) But I still have to add more tests anyways, so I'll look into adding some pylint disables for those no-member issues.

chrisjsewell · 2021-02-17T11:01:46Z

For the record, that pre-commit issue is not from files changed in this PR.

just investigating

chrisjsewell · 2021-02-17T11:04:28Z

see #4756

sphuber

Thanks @mbercx , concept is looking good, just have some comments about the implementation. Also, it would be good to have some better test coverage once we agree on the implementation and features.

aiida/cmdline/params/options/main.py

aiida/cmdline/commands/cmd_group.py

mbercx · 2022-01-26T12:35:17Z

Thanks @sphuber! I've added the -a / --all option and increased the test coverage. Let's see if I managed to sufficiently appease the codecov gods...

sphuber

Thanks @mbercx some final requests to the behavior.

aiida/cmdline/commands/cmd_group.py

@sphuber

I'm ok with this now (although I still find the "failing in the middle of the remove_nodes/add_nodes problematic, I don't really have a good solution either xD), so I'll leave the final aproval to @sphuber.

Add an extra subcommand `move-nodes` to `verdi group` that moves nodes from a source group to a target group, instead of having to use the `remove-nodes` and `add-nodes` subcommands in order to achieve this. Similar to the `remove-nodes` and `add-nodes` commands, the move can be made without prompts by adding the `--force` flag, and the nodes are presented as a list of arguments. Add corresponding test method to the `TestVerdiGroup` class of the test_group.py suite.

Here we add several warnings and failures depending on the provided input: Warnings: * If some of the specified nodes aren't in the source group. * If some of the specified nodes are in the target group. Critical Failures: * If the source and target group are the same. * If no node identifiers are provided to move. * If none of the specified nodes are in the source group. All of these can be skipped by using the `--force` flag.

for more information, see https://pre-commit.ci

mbercx · 2022-01-26T14:13:22Z

Thanks for the comments @sphuber! I agree with the revised logic, I think originally I was for some reason very focused on --force not returning any output for some reason. 🤷

Here is an updated example of the usage:

For two groups:

$ verdi group list -C
Report: To show groups of all types, use the `-a/--all` option.
  PK  Label        Type string    User                Node count
----  -----------  -------------  ----------------  ------------
   2  dummygroup1  core           mbercx@gmail.com             2
   3  dummygroup2  core           mbercx@gmail.com             2

Which each contain two nodes:

$ verdi group show 2
-----------------  ----------------
Group label        dummygroup1
Group type_string  core
Group description  <no description>
-----------------  ----------------
# Nodes:
  PK  Type             Created
----  ---------------  --------------
 279  CalculationNode  3h:30m:37s ago
 280  Int              3h:30m:37s ago

$ verdi group show 3
-----------------  ----------------
Group label        dummygroup2
Group type_string  core
Group description  <no description>
-----------------  ----------------
# Nodes:
  PK  Type    Created
----  ------  --------------
 280  Int     3h:30m:40s ago
 281  Bool    3h:30m:40s ago

Trying to move nodes to the same group:

$ verdi group move-nodes -s dummygroup1 -t dummygroup1
Critical: Source and target group are the same: Group<dummygroup1>.

Not specifying any nodes or --all:

$ verdi group move-nodes -s dummygroup1 -t dummygroup2
Critical: Neither NODES or the `-a, --all` option was specified.

No nodes you want to move aren't in the source group:

$ verdi group move-nodes -s dummygroup1 -t dummygroup2 281
Critical: None of the specified nodes are in Group<dummygroup1>.

Some nodes you want to move aren't in the source group:

$ verdi group move-nodes -s dummygroup1 -t dummygroup2 1 2 3 279
Warning: 3 nodes with PK {1, 2, 3} are not in Group<dummygroup1>.
Are you sure you want to move 1 nodes from Group<dummygroup1> to Group<dummygroup2>? [y/N]:

The node you want to move is in the source group, but also already in the target group:

$ verdi group move-nodes -s dummygroup1 -t dummygroup2 280
Warning: 1 nodes with PK {280} are already in Group<dummygroup2>. These will still be removed from Group<dummygroup1>.
Are you sure you want to move 1 nodes from Group<dummygroup1> to Group<dummygroup2>? [y/N]:

Mixing it up:

$ verdi group move-nodes -s dummygroup1 -t dummygroup2 1 2 3 280 279
Warning: 3 nodes with PK {1, 2, 3} are not in Group<dummygroup1>.
Warning: 1 nodes with PK {280} are already in Group<dummygroup2>. These will still be removed from Group<dummygroup1>.
Are you sure you want to move 2 nodes from Group<dummygroup1> to Group<dummygroup2>? [y/N]:

ramirezfranciscof previously requested changes Oct 9, 2020

View reviewed changes

aiida/cmdline/commands/cmd_group.py Show resolved Hide resolved

tests/cmdline/commands/test_group.py Outdated Show resolved Hide resolved

mbercx mentioned this pull request Oct 12, 2020

Removing nodes that aren't in Group raises no warning #4438

Closed

mbercx force-pushed the feat/4426/group_move_nodes branch from 7544278 to fcc7ba3 Compare February 16, 2021 19:20

mbercx force-pushed the feat/4426/group_move_nodes branch from fcc7ba3 to 7bef6a3 Compare February 16, 2021 19:21

mbercx requested a review from yakutovicha February 16, 2021 19:21

mbercx requested a review from ramirezfranciscof February 16, 2021 19:36

yakutovicha reviewed Feb 16, 2021

View reviewed changes

aiida/cmdline/commands/cmd_group.py Outdated Show resolved Hide resolved

ramirezfranciscof self-assigned this Mar 30, 2021

mbercx force-pushed the feat/4426/group_move_nodes branch 2 times, most recently from b128aaf to f278aac Compare January 26, 2022 08:16

mbercx changed the title ~~Add verdi group move-nodes command~~ ✨ NEW: Add verdi group move-nodes command Jan 26, 2022

sphuber requested changes Jan 26, 2022

View reviewed changes

aiida/cmdline/commands/cmd_group.py Outdated Show resolved Hide resolved

aiida/cmdline/commands/cmd_group.py Outdated Show resolved Hide resolved

aiida/cmdline/commands/cmd_group.py Outdated Show resolved Hide resolved

aiida/cmdline/commands/cmd_group.py Outdated Show resolved Hide resolved

mbercx and others added 5 commits January 26, 2022 15:06

[pre-commit.ci] auto fixes from pre-commit.com hooks

41c55f9

for more information, see https://pre-commit.ci

Apply reviewer suggestions

d50cf90

apply reviewer suggestions

d1d98c9

mbercx force-pushed the feat/4426/group_move_nodes branch from aa73e5e to d1d98c9 Compare January 26, 2022 14:06

sphuber approved these changes Jan 26, 2022

View reviewed changes

sphuber merged commit 4329551 into aiidateam:develop Jan 26, 2022

mbercx deleted the feat/4426/group_move_nodes branch January 26, 2022 14:39

mbercx mentioned this pull request Jan 26, 2022

✨ NEW: _repr_pretty_ representation for ProcessBuilder #4970

Merged

chrisjsewell mentioned this pull request Mar 12, 2022

🔀 MERGE: Release v2.0.0b1 #5426

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ NEW: Add `verdi group move-nodes` command #4428

✨ NEW: Add `verdi group move-nodes` command #4428

mbercx commented Oct 7, 2020

ramirezfranciscof left a comment

mbercx commented Oct 9, 2020

mbercx commented Feb 16, 2021

mbercx commented Feb 16, 2021

codecov bot commented Feb 16, 2021 •

edited

Loading

yakutovicha left a comment •

edited

Loading

mbercx commented Feb 17, 2021

yakutovicha commented Feb 17, 2021

yakutovicha commented Feb 17, 2021

mbercx commented Feb 17, 2021

chrisjsewell commented Feb 17, 2021

chrisjsewell commented Feb 17, 2021

sphuber left a comment

mbercx commented Jan 26, 2022

sphuber left a comment

mbercx commented Jan 26, 2022

✨ NEW: Add verdi group move-nodes command #4428

✨ NEW: Add verdi group move-nodes command #4428

Conversation

mbercx commented Oct 7, 2020

ramirezfranciscof left a comment

Choose a reason for hiding this comment

mbercx commented Oct 9, 2020

mbercx commented Feb 16, 2021

mbercx commented Feb 16, 2021

codecov bot commented Feb 16, 2021 • edited Loading

Codecov Report

yakutovicha left a comment • edited Loading

Choose a reason for hiding this comment

mbercx commented Feb 17, 2021

yakutovicha commented Feb 17, 2021

yakutovicha commented Feb 17, 2021

mbercx commented Feb 17, 2021

chrisjsewell commented Feb 17, 2021

chrisjsewell commented Feb 17, 2021

sphuber left a comment

Choose a reason for hiding this comment

mbercx commented Jan 26, 2022

sphuber left a comment

Choose a reason for hiding this comment

mbercx commented Jan 26, 2022

✨ NEW: Add `verdi group move-nodes` command #4428

✨ NEW: Add `verdi group move-nodes` command #4428

codecov bot commented Feb 16, 2021 •

edited

Loading

yakutovicha left a comment •

edited

Loading