use uutils/coreutils cp command in place of nushell's cp command #10097

dmatos2012 · 2023-08-22T22:32:24Z

Description

Hi. Basically, this is a continuation of the work that @fdncred started. Given some nice discussions on #9463 , and merged uutils PR from @tertsdiepraam we have decided to give the cp command the crawl stage as it was named.

Note

Given that the uutils crate has not made the release for the merged PR, just make sure you checkout latest and put it in the required place to make this PR work.

The aim of this PR is for is to see how to move forward using uutils crate. In order to getting this started, I have made the current nushell cp tests pass along with some extra ones I copied over from the uutils repo.

With all of that being said, things that would be nice to decide, and keep working on:

Crawl:

Handling of certain named flags, with their long and short forms(e.g. --update, --reflink, --preserve, etc), and using default values. Maybe -u can already have a default_missing_value.
Should we maybe just support one single option switch flags (see --backup in code) as a contrast to the other named args.
Complete test coverage from uutils. They had > 100 tests, and I could only port like 12 as they are a bit time consuming given they cannot be straight up copy pasted. Maybe we do not need all >100, but maybe the more relevant to what we want.
Refactor this code

Walk:

Non fatal errors on copy from utils. Currently it just sends it to stdout but errors have no span
Better integration

An added possibility is the addition of SyntaxShape::OneOf() for Named arguments which was briefly mentioned in the discord server, but that is still to be decided. This could greatly improve some of the integration. This would enable something like cp --preserve [all timestamp] or cp --preserve all to both work.

I did not want to keep holding on this, and wait till I was happy with the code because I think its nice if everyone can start up and suggest refactors, but the main important part now was getting it out the door, as if I take my sweet time this will take way longer 😛

User-Facing Changes

Tests + Formatting

Make sure you've run and fixed any issues with these commands:

cargo fmt --all -- --check to check standard code formatting (cargo fmt --all` applies these changes)
cargo clippy --workspace -- -D warnings -D clippy::unwrap_used` to check that you're using the standard code style
cargo test --workspace` to check that all tests pass
cargo run -- -c "use std testing; testing run-tests --path crates/nu-std"` to run the tests for the standard library

Note
from nushell you can also use the toolkit as follows
use toolkit.nu  # or use an `env_change` hook to activate it automatically
toolkit check pr

-->

After Submitting

dmatos2012 · 2023-08-22T22:42:48Z

Forgot: For the live of me I could not figure out how to add a env variable to the test on test_cp_custom_backup_suffix_via_env(), so its currently the only one failing. If i do the test manually, and add the env var manually, it works just fine, but apparently the test doesnt detect the env :(

fdncred · 2023-08-22T22:53:06Z

Thanks for working on this @dmatos2012!! We appreciate your efforts. Can you try to get the CI green? Just comment out the env test if that's needed. I think it'll be easier to continue with it being green.

At first glance, I just go, "wow, look at all the parameter checking code". 98% of all this code is just parameter checking. In the coming days, I'm sure the core-team will look into this closer and suggestion changes.

fdncred · 2023-08-22T23:20:27Z

I'm also getting these errors. Not sure if it's because of a preliminary BackupMode attempt or if it's because we need @tertsdiepraam to make something else public?

error[E0603]: module `mods` is private
  --> crates/nu-command/src/filesystem/ucp.rs:11:13
   |
11 | use uucore::mods::backup_control::BackupMode;
   |             ^^^^ private module
   |
note: the module `mods` is defined here
  --> /Users/fdncred/src/coreutils/src/uucore/src/lib/lib.rs:20:1
   |
20 | mod mods; // core cross-platform modules
   | ^^^^^^^^

error[E0603]: module `mods` is private
  --> crates/nu-command/src/filesystem/ucp.rs:12:13
   |
12 | use uucore::mods::update_control::UpdateMode;
   |             ^^^^ private module
   |
note: the module `mods` is defined here
  --> /Users/fdncred/src/coreutils/src/uucore/src/lib/lib.rs:20:1
   |
20 | mod mods; // core cross-platform modules
   | ^^^^^^^^

tertsdiepraam

Excellent! I'll review this more from a design and integration perspective.

I think choosing to make --backup not take any arguments is a pretty good idea. In general, I'd say that the goal here is a simplified, more nushell-style interface to cp. We don't necessesarily have to take our inspiration from GNU either. For example, FreeBSD cp has only switches and no arguments with values and that seems to work all right for them.¹

They also have just -p, for example, which just preserves all attributes.So, I think we could even decide to make --preserve and --update also just switches for instance.

Another good idea in here is that you make conflicting arguments just error. In uutils, I begrudingly make everything override because that's GNU's behaviour, but here we're free to just make things error :smile

https://man.freebsd.org/cgi/man.cgi?query=cp&sektion=1&manpath=FreeBSD ↩

tertsdiepraam · 2023-08-22T23:07:23Z

crates/nu-command/src/filesystem/ucp.rs

+            progress_bar: progress,
+        };
+
+        // For enabling easy `crawl` of cp command, we need to strip current directory


Does cp not support absolute paths? That would be a pretty serious bug!

tertsdiepraam · 2023-08-22T23:09:09Z

crates/nu-command/Cargo.toml

@@ -90,6 +90,8 @@ toml = "0.7"
 unicode-segmentation = "1.10"
 ureq = { version = "2.7", default-features = false, features = ["charset", "gzip", "json", "native-tls"] }
 url = "2.2"
+uu_cp = { path = "../../../coreutils/src/uu/cp"}
+uucore = { path = "../../../coreutils/src/uucore"}


What is this used for? I hoped we could do this without directly depending on anything but uu_cp (even though uucore would still be a transitive dependency).

I tried to compile with and without uucore. This screenshot is with it, but also trying to reference the modes directly.

tertsdiepraam · 2023-08-22T23:15:20Z

crates/nu-command/src/filesystem/ucp.rs

+                }
+            };
+        }
+        //                 // uucore::error::set_exit_code(EXIT_ERR);


Remember to clean this up :)

tertsdiepraam · 2023-08-22T23:16:35Z

crates/nu-command/src/filesystem/ucp.rs

+                // ShellError
+                // _ => uucore::macros::show_error!("{}", error),
+                _ => {
+                    return Err(ShellError::GenericError(


We could look into integrating UResult/UError with the nu system for future utils to make this easier.

tertsdiepraam · 2023-08-22T23:21:01Z

crates/nu-command/src/filesystem/ucp.rs

+};
+
+/// Returns tuple of (Source paths, Target)
+pub fn parse_path_args(


I'd say we're being too compatible here. I think the way to go is to make it super simple: parse the paths, pop the last one and check that both are non-empty. We can then ignore --target-dir

tertsdiepraam · 2023-08-22T23:25:28Z

Not sure if it's because of a preliminary BackupMode attempt or if it's because we need @tertsdiepraam to make something else public?

It's both, sort of. I think @dmatos2012 used the wrong path to import it, because it should be public, since cp can use it. But I also think that uu_cp should re-export it.

fdncred · 2023-08-23T02:10:22Z

Just chiming in again on these comments.

I think choosing to make --backup not take any arguments is a pretty good idea.

Agreed.

In general, I'd say that the goal here is a simplified, more nushell-style interface to cp.

Agreed.

We don't necessarily have to take our inspiration from GNU either.

Agreed.

Another thought I keep having is that nushell does not have to support every parameter uu_cp supports. In fact, the nushell maintainers, have done their best to not have dozens of parameters on internal commands. I assume that some of the core-team will not approve this PR with, if I counted correctly, 29 parameters. I'm not saying that to be critical, but I'm just predicting that we'll want less, not more. The trick will be to decide which parameters we'll want to keep. I'm also guessing that this less-is-more approach will apply to any future commands as well.

tertsdiepraam · 2023-08-23T06:51:46Z

@fdncred looks like we're very much on the same page. I love how simple nushells commands tend to be. Maybe to help @dmatos2012 out, could you make a list of arguments that you and the core team think are important?

dmatos2012 · 2023-08-24T19:36:21Z

Hey, so I think given the feedback you folks gave, I think I will start by making the change for --update and --preserve as both switches, and go from there. Additionally , I think @tertsdiepraam idea of perhaps presenting a list of arguments that for sure are very important to the Nu core Team @fdncred would be very interesting, so that we can start by removing some of the extra ones

. In my opinion, but it could be because I am not a hardcore cp person, Id say the --reflink , --sparse are good candidates to be removed, as they also have lots of repetitive arg parsing. :)

fdncred · 2023-08-24T19:51:18Z

Thanks @dmatos2012, and some files may be missing or not configured correctly which caused the CI to puke.

The @nushell/core-team talked about listing the parameters that are important to us on the team meeting yesterday. I plan on making a list for the team to look at.

One last thing we talked about yesterday, we definitely want to try and reduce dependencies if possible. I think we can remove uucore if @tertsdiepraam makes the BackupMode and UpdateMode public. We, at nushell, are always very concerned about the number of dependencies we add and there are quite a few new ones right now. I wouldn't worry too much about this yet, but I wanted to bring it up so y'all understand.

tertsdiepraam · 2023-08-24T20:12:39Z

The @nushell/core-team talked about listing the parameters that are important to us on the team meeting yesterday. I plan on making a list for the team to look at.

Excellent!

One last thing we talked about yesterday, we definitely want to try and reduce dependencies if possible. I think we can remove uucore if @tertsdiepraam makes the BackupMode and UpdateMode public.

It would be quite hard for us to remove uucore at this stage. Maybe over time uucore will be split up into multiple crates. That would be great in my opinion, but I've not discussed this with the other maintainers and would require quite a bit of work. My idea for now was to only make it a transitive dependency, which of course means that it would still be there, but at least you don't have to bump the version number. 😄

I agree though that it's quite big, so I'll open an issue to at least put some more dependencies behind feature flags, because we definitely also do not need every dependency for every util. (uutils/coreutils#5202)

Another problem I see dependency-wise is clap, which would be compiled but not used at all, because you have your own parsing. I'll open up an issue on uutils to design some way of avoiding that. (uutils/coreutils#5203)

Edit: The issues are up. I added the links to the text above.

Edit 2: I'd also love to call via Discord sometime to catch these kinds of issues a bit quicker. Maybe I could join one of the core team calls or we could set up a separate call. Other uutils maintainers and dmatos might also want to join.

dmatos2012 · 2023-08-24T21:27:17Z

Edit 2: I'd also love to call via Discord sometime to catch these kinds of issues a bit quicker. Maybe I could join one of the core team calls or we could set up a separate call. Other uutils maintainers and dmatos might also want to join.

Id be up for joining!

And unless I am not getting it right, isnt it expected that the CI will fail (for now)? Its currently pointing to my local path of coreutils since the new release has not been made yet. And btw the small changes we discussed were already made!

fdncred · 2023-08-24T21:43:06Z

It would be quite hard for us to remove uucore at this stage.

I just meant remove it from the nushell Cargo.toml. It seemed to be able to do that when I was testing a yesterday.

Edit 2: I'd also love to call via Discord sometime to catch these kinds of issues a bit quicker. Maybe I could join one of the core team calls or we could set up a separate call. Other uutils maintainers and dmatos might also want to join.

Our team meetings are public every Wednesday at 3pm Central time. Anyone is welcome. If we want to have a special section in the meeting about uutils, I just need some bullet topics that we want to discuss. We're very informal.

And unless I am not getting it right, isnt it expected that the CI will fail (for now)? Its currently pointing to my local path of coreutils since the new release has not been made yet. And btw the small changes we discussed were already made!

@dmatos2012 maybe you're right? but couldn't we point it to the particular git branch in uutils/coreutils and make it work?

dmatos2012 · 2023-08-24T21:48:03Z

@dmatos2012 maybe you're right? but couldn't we point it to the particular git branch in uutils/coreutils and make it work?

Ahh, smart! Yeah I will try that tomorrow, seems doable but now its time to sleep in Europe lol.

tertsdiepraam · 2023-08-24T22:04:59Z

Our team meetings are public every Wednesday at 3pm Central time. Anyone is welcome. If we want to have a special section in the meeting about uutils, I just need some bullet topics that we want to discuss. We're very informal.

Cool! I might join on Wednesday. I'll let you know and we can discuss topics beforehand. I'll also tell the other uutils maintainers.

And btw the small changes we discussed were already made!

Very nice!

fdncred · 2023-08-27T12:11:02Z

Any progress on this @dmatos2012? Are there blockers that you need help with? Are you waiting for anyone to do something or you're just not finding the time yet?

fdncred · 2023-08-27T12:36:10Z

I've created a voting spreadsheet for the @nushell/core-team members to vote on parameters to keep. @dmatos2012 and @tertsdiepraam feel free to add a column to collect your votes as well. I don't really understand all the parameters but I put some votes in to show what I'm expecting.

https://docs.google.com/spreadsheets/d/1aFXAg2IkHFCz289QYBroSn9XK_80lV3y7KEZxzeHw2c/edit?usp=sharing

tertsdiepraam · 2023-08-27T12:52:25Z

Looks good! I'll add a column! I'd also suggest a column with the BSD set of options for reference.

The link only seems to get me view access at the moment. I've sent you a request for edit access.

fdncred · 2023-08-27T13:05:05Z

I'd also suggest a column with the BSD set of options for reference.

@tertsdiepraam feel free to augment. I updated the links, sorry, forgot to hit a button. :)

amtoine · 2023-08-27T13:46:27Z

voted in the sheet 😉

dmatos2012 · 2023-08-27T13:54:48Z

Any progress on this @dmatos2012? Are there blockers that you need help with? Are you waiting for anyone to do something or you're just not finding the time yet?

Hi @fdncred , I now made the CI happy, since I had made the mistake of changing some of the cp tests from before, forgetting they would only be working under the --features nuuu 😞 but now its fixed.

As far as I know, given that now CI is happy, now I am waiting for the wednesday meeting to define what flags we would be interested on, and then change the code to reflect that :)

fdncred · 2023-08-27T19:15:10Z

@tertsdiepraam Looks like there are 3 things that are blocking us from removing uucore as a dependency in our nu-command Cargo.toml. BackupMode, UpdateMode and set_exit_code. I tried to compile without uucore in the nu-command Cargo.toml and with this change to the ucp.rs. What is your opinion on how we should handle this?

// use uucore::backup_control::{self, BackupMode};
// use uucore::update_control::{self, UpdateMode};
use uu_cp::{BackupMode, UpdateMode};

One school of thought is to make these public so that uu_cp can find them.
Another school of thought is, maybe backup/update modes won't matter if we don't use those flags/options.
Yet another is, since we're going to add more uu_ crates in future PRs, we're going to need uucore anyway so it shouldn't matter much.

tertsdiepraam · 2023-08-27T19:56:02Z

Interesting. I don't think set_exit_code works correctly now, because it sets an AtomicI32 that we in read in the code generated by the uumain macro if I recall correctly. We don't use that macro here so that value won't be read. @dmatos2012 do we have a test case for that path?

For the other two, my preference is option 1, which I could make a PR for very quickly.

Option 1 gives a full API to access uu_cp both for nu and possibly other projects. Option 2 doesn't quite work, because even if --update is just a switch, nu still has to use that enum to tell uu_cp about that flag. Option 3 would be fine, but ideally all the uutils crates should be usable independently.

Shall I make a PR for option 1?

Reflect those on tests & code

Update uutils with progress bar fix

Rename current cp to CpOld

apply suggested changes

Rename commands again

dmatos2012 · 2023-09-08T06:55:22Z

So I used some time before work 😛 , and proceeded to do as @fdncred indicated in the last comment. I also agree its a very good idea as it gives us some real life usage. Most likely, it wont be the only uncovered bug, so we could keep adding tests in the meantime.

I additionally split the tests now into cp.rs and ucp.rs. Basically ucp.rs contains the original cp.rs tests with modifications and additions, and the cp.rs is the original. Even though there is duplication now, this hopefully is only for the short term so hope is ok. If not, I can move back.

fdncred · 2023-09-08T11:51:48Z

Thanks @dmatos2012. I'm trying to get a thumbs up/down on this right now. Hopefully we can land this today if it's approved by the nushell core-team.

fdncred · 2023-09-08T18:57:35Z

The nushell core-team agreed on this strategy. Let's go!!!!!

dmatos2012 · 2023-09-08T19:11:51Z

The nushell core-team agreed on this strategy. Let's go!!!!!

Let's goo!!! Very excited for next commands to port and keep improving on the current ones. Thanks for all the guidance and patience here, it's so lovely to contribute here @tertsdiepraam @fdncred

tertsdiepraam · 2023-09-08T19:29:57Z

Whooo!!! 🚀🚀

I'm already excited for the next util, do we already know which one that's gonna be? 😄 Great work here @dmatos2012 and thanks for the coordination and guidance @fdncred!

fdncred · 2023-09-08T19:41:56Z

@tertsdiepraam I think the next 2 should be mv and rm.

stormasm · 2023-09-09T03:19:17Z

@tertsdiepraam @dmatos2012
Let me throw my hat in the ring of congratulations for landing this PR as well and say super fantastic job ! You all are amazing...

WindSoilder · 2023-09-09T05:44:05Z

@dmatos2012 @tertsdiepraam Thank you for working on this! Really excited about the new cp!

It's time to make alias cp = ucp

dmatos2012 marked this pull request as draft August 22, 2023 22:32

dmatos2012 mentioned this pull request Aug 22, 2023

cp: make more types public and add more documentation (for nushell) uutils/coreutils#5152

Merged

fdncred added the coreutils-uutils Changes relating to coreutils/uutils label Aug 22, 2023

tertsdiepraam reviewed Aug 22, 2023

View reviewed changes

fdncred mentioned this pull request Aug 24, 2023

use uutils uu_cp and uu_mv #9463

Closed

fdncred changed the title ~~Add crawl stage cp using uutils~~ use uutils/coreutils cp command in place of nushell's cp command Aug 24, 2023

fdncred mentioned this pull request Aug 24, 2023

cp progress bar - cleaner version #8325

Closed

This was referenced Aug 24, 2023

uucore: put more deps behind feature flags (for nushell) uutils/coreutils#5202

Closed

cp: compile without clap (for nushell) uutils/coreutils#5203

Open

dmatos2012 and others added 16 commits September 8, 2023 08:41

Change to coreutils main branch instead of local

9dd78a7

Update tests to make CI happy

775ad2c

Try to fix windows CI error

f0c0a46

remove uucore, merge main

ef2aee5

Reduce number of flags to desired number

98abaf1

Reflect those on tests & code

Refactor slightly and remove parse_path_args

f018bf6

Ignore tests with differing behavior

f97912e

Update uutils with progress bar fix

Replace cp command with uutils cp version

9552696

Rename current cp to CpOld

Update uutils to latest release

bab2761

Fix example failing test

9606083

remove old uu_cp comments

e7f6c31

Add extra test to check single source

320a476

apply suggested changes

Fix cp tests numbering

9ce689d

ignore an error

397a31c

removed by mistake, undo

297b48e

Separate tests for cp and ucp

46f0f39

Rename commands again

dmatos2012 force-pushed the uutils_ucp branch from 650d74a to 46f0f39 Compare September 8, 2023 06:50

Clippy lint

9f91b08

fdncred merged commit fed4233 into nushell:main Sep 8, 2023
20 checks passed

Hofer-Julian mentioned this pull request Oct 4, 2023

Consider relying on uutils/coreutils denoland/deno_task_shell#94

Closed

dmatos2012 deleted the uutils_ucp branch October 24, 2023 20:15

fdncred mentioned this pull request Jan 16, 2024

add uutils/coreutils commands into nushell #11549

Open

16 tasks

use uutils/coreutils cp command in place of nushell's cp command #10097

use uutils/coreutils cp command in place of nushell's cp command #10097

Conversation

dmatos2012 commented Aug 22, 2023

Description

User-Facing Changes

Tests + Formatting

After Submitting

dmatos2012 commented Aug 22, 2023

fdncred commented Aug 22, 2023

fdncred commented Aug 22, 2023

tertsdiepraam left a comment

Choose a reason for hiding this comment

Footnotes

tertsdiepraam Aug 22, 2023

Choose a reason for hiding this comment

tertsdiepraam Aug 22, 2023

Choose a reason for hiding this comment

fdncred Aug 23, 2023

Choose a reason for hiding this comment

tertsdiepraam Aug 22, 2023

Choose a reason for hiding this comment

tertsdiepraam Aug 22, 2023

Choose a reason for hiding this comment

tertsdiepraam Aug 22, 2023

Choose a reason for hiding this comment

tertsdiepraam commented Aug 22, 2023

fdncred commented Aug 23, 2023

tertsdiepraam commented Aug 23, 2023

dmatos2012 commented Aug 24, 2023

fdncred commented Aug 24, 2023

tertsdiepraam commented Aug 24, 2023 • edited

dmatos2012 commented Aug 24, 2023 • edited

fdncred commented Aug 24, 2023

dmatos2012 commented Aug 24, 2023 • edited

tertsdiepraam commented Aug 24, 2023 • edited

fdncred commented Aug 27, 2023

fdncred commented Aug 27, 2023 • edited

tertsdiepraam commented Aug 27, 2023

fdncred commented Aug 27, 2023

amtoine commented Aug 27, 2023

dmatos2012 commented Aug 27, 2023 • edited

fdncred commented Aug 27, 2023 • edited

tertsdiepraam commented Aug 27, 2023

dmatos2012 commented Sep 8, 2023

fdncred commented Sep 8, 2023

fdncred commented Sep 8, 2023 • edited

dmatos2012 commented Sep 8, 2023

tertsdiepraam commented Sep 8, 2023

fdncred commented Sep 8, 2023

stormasm commented Sep 9, 2023 • edited

WindSoilder commented Sep 9, 2023

tertsdiepraam commented Aug 24, 2023 •

edited

dmatos2012 commented Aug 24, 2023 •

edited

dmatos2012 commented Aug 24, 2023 •

edited

tertsdiepraam commented Aug 24, 2023 •

edited

fdncred commented Aug 27, 2023 •

edited

dmatos2012 commented Aug 27, 2023 •

edited

fdncred commented Aug 27, 2023 •

edited

fdncred commented Sep 8, 2023 •

edited

stormasm commented Sep 9, 2023 •

edited