How to redirect stderr but continue to pipe stdout?

[bobhy]

Question

I'm having trouble invoking an external command and redirecting stderr (to /dev/null) while piping stdout to the next step.
Maybe there's some other kind of redirection I should be using? Maybe it's a bug?

Sample:

> sudo apt update  | filter {|line| $line | str contains --ignore-case "can be upgraded" } 

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

╭───┬──────────────────────────────────────────────────────────────────────╮
│ 0 │ 6 packages can be upgraded. Run 'apt list --upgradable' to see them. │
│   │                                                                      │
╰───┴──────────────────────────────────────────────────────────────────────╯

Almost what I want, except for the ugly WARNING that didn't flow into the pipe.

(try to) Fix it by redirecting stderr:

> sudo apt update  err> /dev/null | filter {|line| $line | str contains --ignore-case "can be upgraded" } 
Hit:1 http://us.archive.ubuntu.com/ubuntu lunar InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu lunar-updates InRelease                                                      
Hit:3 http://packages.microsoft.com/repos/code stable InRelease                                                        
Hit:4 http://us.archive.ubuntu.com/ubuntu lunar-backports InRelease                                                    
Hit:5 https://packages.microsoft.com/repos/edge stable InRelease                                                       
Hit:6 http://security.ubuntu.com/ubuntu lunar-security InRelease           
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
6 packages can be upgraded. Run 'apt list --upgradable' to see them.

Does suppress the WARNING, but now filter isn't getting any input...

At first I though this had something to do with sudo, but you can repro with nonprivileged external dir command

> dir -l .  argle  | filter {|x| $x | str contains Music}
dir: cannot access 'argle': No such file or directory
╭───┬────────────────────────────────────────────────────────╮
│ 0 │ .:                                                     │
│   │ total 89                                               │
│   │ drwxr-xr-x 2 bobhy bobhy   2 Aug 18 21:12 Desktop      │
│   │ drwxr-xr-x 2 bobhy bobhy   3 Sep  6 14:20 Documents    │
│   │ drwxr-xr-x 2 bobhy bobhy  25 Sep  6 23:47 Downloads    │
│   │ drwxr-xr-x 2 bobhy bobhy   2 Aug 18 21:12 Music        │
│   │ drwxr-xr-x 3 bobhy bobhy   3 Aug 25 13:18 Pictures     │
│   │ drwxr-xr-x 2 bobhy bobhy   2 Aug 18 21:12 Public       │
│   │                                                        │
╰───┴────────────────────────────────────────────────────────╯

The missing directory error is reported to stderr as in the previous case.

I don't quite get why each newline-delimited line of output wasn't broken into a separate line item, like it seems to have done with sudo apt, but never mind: the key thing is filter returned a list with one element, as desired.

> dir -l .  argle err> /dev/null  | filter {|x| $x | str contains Music}
.:
total 89
drwxr-xr-x 2 bobhy bobhy   2 Aug 18 21:12 Desktop
drwxr-xr-x 2 bobhy bobhy   3 Sep  6 14:20 Documents
drwxr-xr-x 2 bobhy bobhy  25 Sep  6 23:47 Downloads
drwxr-xr-x 2 bobhy bobhy   2 Aug 18 21:12 Music
drwxr-xr-x 3 bobhy bobhy   3 Aug 25 13:18 Pictures
drwxr-xr-x 2 bobhy bobhy   2 Aug 18 21:12 Public

Note that the dir output isn't being piped into filter,at all, apparently.

Additional context and details

No response

[fdncred]

I can only think of 3 things that I'd try.

1. put the redirection at the end just to see what happens like this sudo apt update | filter {|line| $line | str contains --ignore-case "can be upgraded" } err> /dev/null. Seems illogical, but I'm curious.
2. use complete like do -i {sudo apt update | filter ...} | complete | get stdout, which isn't the same but may work.
3. This is what is used in atuin run-external --redirect-stderr atuin search "--interactive" | complete | $in.stderr

I agree with you that sudo apt update err> /dev/null | filter {|line| $line | str contains --ignore-case "can be upgraded" } seems like it should work. I'm not sure what's going on.

@WindSoilder, any ideas about this?

[horasal]

I'm not sure but a quick look through eval.rs, it may be:

• if any redirection exists, all the output will just be ignored and not piped.
• apt is tiggered through run-external but it does not always cache stdout to pipeline

workround can be use run-external with --redirect-stderr and --redirect-stdout:

❯ cat test.sh
#!/bin/bash

echo this is stderr message >> /dev/stderr
echo stdout message
echo stdout message line 2

❯ run-external ./test.sh --redirect-stdout --redirect-stderr  | lines
╭───┬───────────────────────╮
│ 0 │ stdout message        │
│ 1 │ stdout message line 2 │
╰───┴───────────────────────╯

the following quick patch makes err> pipe stdout (but it breaks out> :(

diff --git a/crates/nu-engine/src/eval.rs b/crates/nu-engine/src/eval.rs
index 462dbb347..cea130517 100644
--- a/crates/nu-engine/src/eval.rs
+++ b/crates/nu-engine/src/eval.rs
@@ -203,7 +203,6 @@ fn eval_external(

     match redirect_target {
         RedirectTarget::Piped(redirect_stdout, redirect_stderr) => {
-            if redirect_stdout {
                 call.add_named((
                     Spanned {
                         item: "redirect-stdout".into(),
@@ -211,10 +210,7 @@ fn eval_external(
                     },
                     None,
                     None,
-                ))
-            }
-
-            if redirect_stderr {
+                ));
                 call.add_named((
                     Spanned {
                         item: "redirect-stderr".into(),
@@ -223,7 +219,6 @@ fn eval_external(
                     None,
                     None,
                 ))
-            }
         }
         RedirectTarget::CombinedPipe => call.add_named((
             Spanned {
@@ -746,10 +741,11 @@ pub fn eval_element_with_input(
                     PipelineData::ExternalStream { exit_code, .. } => exit_code.take(),
                     _ => None,
                 };
-                let input = match (redirection, input) {
+                let (input_to_save, output_stdout, output_stderr) = match (redirection, input) {
                     (
                         Redirection::Stderr,
                         PipelineData::ExternalStream {
+                            stdout,
                             stderr,
                             exit_code,
                             span,
@@ -757,15 +753,34 @@ pub fn eval_element_with_input(
                             trim_end_newline,
                             ..
                         },
-                    ) => PipelineData::ExternalStream {
+                    ) => (PipelineData::ExternalStream {
                         stdout: stderr,
                         stderr: None,
                         exit_code,
                         span,
                         metadata,
                         trim_end_newline,
-                    },
-                    (_, input) => input,
+                    }, stdout, None),
+                    (
+                        Redirection::Stdout,
+                        PipelineData::ExternalStream {
+                            stdout,
+                            stderr,
+                            exit_code,
+                            span,
+                            metadata,
+                            trim_end_newline,
+                            ..
+                        },
+                    ) => (PipelineData::ExternalStream {
+                        stdout: stdout,
+                        stderr: None,
+                        exit_code,
+                        span,
+                        metadata,
+                        trim_end_newline,
+                    }, None, stderr),
+                    (_, input) => (input, None, None),
                 };

                 if let Some(save_command) = engine_state.find_decl(b"save", &[]) {
@@ -798,7 +813,7 @@ pub fn eval_element_with_input(
                             redirect_stderr: false,
                             parser_info: HashMap::new(),
                         },
-                        input,
+                        input_to_save,
                     )
                     .map(|_| {
                         // save is internal command, normally it exists with non-ExternalStream
@@ -806,8 +821,8 @@ pub fn eval_element_with_input(
                         // So nu handles exit_code correctly
                         (
                             PipelineData::ExternalStream {
-                                stdout: None,
-                                stderr: None,
+                                stdout: output_stdout,
+                                stderr: output_stderr,
                                 exit_code,
                                 span: *span,
                                 metadata: None,

[WindSoilder]

Sorry, I don't have much idea, I can try to look into this when I get time