detect columns fails to parse cell contents that include spaces

[waldyrious]

Describe the bug

Using the example output from the docker ps documentation:

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
673394ef1d4c        busybox             "top"               47 seconds ago      Up 45 seconds                           nostalgic_shockley
d85756f57265        busybox             "top"               52 seconds ago      Up 51 seconds                           high_albattani

Here's how detect columns parses it:

> cat ~/docker-ps.txt | detect columns
───┬──────────────┬──────────────┬─────────┬─────────┬─────────┬─────────┬───────┬────────────────────
 # │  CONTAINER   │      ID      │  IMAGE  │ COMMAND │ CREATED │ STATUS  │ PORTS │       NAMES        
───┼──────────────┼──────────────┼─────────┼─────────┼─────────┼─────────┼───────┼────────────────────
 0 │ 673394ef1d4c │ 673394ef1d4c │ busybox │ "top"   │ seconds │ seconds │       │ nostalgic_shockley 
 1 │ d85756f57265 │ d85756f57265 │ busybox │ "top"   │ seconds │ seconds │       │ high_albattani     
───┴──────────────┴──────────────┴─────────┴─────────┴─────────┴─────────┴───────┴────────────────────

There are two errors in this output, both related to cell contents that include spaces:

• The CONTAINER ID column is split into two columns, with identical content;
• Cell contents that include spaces (in the CREATED and STATUS columns) is trimmed down to a single word.

How to reproduce

1. Create a ~/docker-ps.txt file with the contents indicated above
2. Run ~/docker-ps.txt | detect columns

Expected behavior

The correct output should be:

> cat ~/docker-ps.txt | detect columns
───┬──────────────┬─────────┬─────────┬────────────────┬───────────────┬───────┬────────────────────
 # │ CONTAINER ID │  IMAGE  │ COMMAND │    CREATED     │    STATUS     │ PORTS │       NAMES        
───┼──────────────┼─────────┼─────────┼────────────────┼───────────────┼───────┼────────────────────
 0 │ 673394ef1d4c │ busybox │ "top"   │ 47 seconds ago │ Up 45 seconds │       │ nostalgic_shockley 
 1 │ d85756f57265 │ busybox │ "top"   │ 52 seconds ago │ Up 51 seconds │       │ high_albattani     
───┴──────────────┴─────────┴─────────┴────────────────┴───────────────┴───────┴────────────────────

Screenshots

No response

Configuration

key	value
version	0.40.0
build_os	linux-x86_64
rust_version	rustc 1.56.1
cargo_version	cargo 1.56.0
pkg_version	0.40.0
build_time	1980-01-01 00:00:00 +00:00
build_rust_channel	release
features	clipboard-cli, ctrlc, dataframe, default, rustyline, term, trash, uuid, which, zip
installed_plugins	binaryview, chart bar, chart line, fetch, from bson, from sqlite, inc, match, post, ps, query json, s3, selector, start, sys, textview, to bson, to sqlite, tree, xpath

Additional context

If it helps, from ssv is able to correctly identify cell contents that include spaces:

> cat ~/docker-ps.txt | from ssv
───┬──────────────┬─────────┬─────────┬────────────────┬───────────────┬────────────────────
 # │ CONTAINER ID │  IMAGE  │ COMMAND │    CREATED     │    STATUS     │       PORTS        
───┼──────────────┼─────────┼─────────┼────────────────┼───────────────┼────────────────────
 0 │ 673394ef1d4c │ busybox │ "top"   │ 47 seconds ago │ Up 45 seconds │ nostalgic_shockley 
 1 │ d85756f57265 │ busybox │ "top"   │ 52 seconds ago │ Up 51 seconds │ high_albattani     
───┴──────────────┴─────────┴─────────┴────────────────┴───────────────┴────────────────────

(There's a separate bug in this output, in that the empty PORTS column isn't properly detected, but I've reported that separately as #4182.)

[fdncred]

detect columns is a relatively new command, but from ssv seems to work fine for this use case | from ssv --aligned-columns

[sophiajt]

detect columns still has this issue on latest main.

@fdncred I wonder if detect columns should work more closely to what | from ssv --aligned-columns does?

[fdncred]

I played around a bit more with this and you can overcome spaces in columns by replacing it like this

❯ open test.txt | str replace "CONTAINER ID" "CONTAINER_ID" | detect columns
╭─#─┬─CONTAINER_ID─┬──IMAGE──┬COMMAND┬─CREATED─┬─STATUS──┬PORTS┬───────NAMES────────╮
│ 0 │ 673394ef1d4c │ busybox │ "top" │ seconds │ seconds │     │ nostalgic_shockley │
│ 1 │ d85756f57265 │ busybox │ "top" │ seconds │ seconds │     │ high_albattani     │
╰───┴──────────────┴─────────┴───────┴─────────┴─────────┴─────┴────────────────────╯

but it still doesn't parse the created and status columns correctly, which I just think may be a bug in detect columns.

from ssv works nicely with this type of data though

❯ open test.txt | from ssv --aligned-columns
╭─#─┬─CONTAINER ID─┬──IMAGE──┬COMMAND┬────CREATED─────┬────STATUS─────┬PORTS┬───────NAMES────────╮
│ 0 │ 673394ef1d4c │ busybox │ "top" │ 47 seconds ago │ Up 45 seconds │     │ nostalgic_shockley │
│ 1 │ d85756f57265 │ busybox │ "top" │ 52 seconds ago │ Up 51 seconds │     │ high_albattani     │
╰───┴──────────────┴─────────┴───────┴────────────────┴───────────────┴─────┴────────────────────╯

One of the things I like about having both of these commands is that if one fails we have the other one to try. They work differently and I think that's good. I'm not sure if we should have 2 commands like this or not. I suspect it would be difficult to merge them.

I've also explored column -t and the tabulate external commands which both have different ways of handling this, but also could help with trying to solve these types of problems. Neither does very well, but I'm not providing any parameters to either.

open test.txt | ^column -t 
CONTAINER     ID       IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES
673394ef1d4c  busybox  "top"  47       seconds  ago     Up     45     seconds  nostalgic_shockley
d85756f57265  busybox  "top"  52       seconds  ago     Up     51     seconds  high_albattani

open test.txt | tabulate 
[src/main.rs:113] &args = Args {
    truncate: None,
    ratio: 1.0,
    lines: 1000,
    include_cols: None,
    exclude_cols: None,
    delim: " \t",
    output_delim: "  ",
    strict_delim: false,
    online: false,
    print_info: false,
}
CONTAINER     ID       IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES
673394ef1d4c  busybox  "top"  47       seconds  ago     Up     45     seconds  nostalgic_shockley
d85756f57265  busybox  "top"  52       seconds  ago     Up     51     seconds  high_albattani

[fdncred]

After the latest changes to the detect columns algorithm.

❯ 'CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
❯❯❯ 673394ef1d4c        busybox             "top"               47 seconds ago      Up 45 seconds                           nostalgic_shockley
❯❯❯ d85756f57265        busybox             "top"               52 seconds ago      Up 51 seconds                           high_albattani' | detect columns
╭─#─┬─CONTAINER ID─┬──IMAGE──┬COMMAND┬────CREATED─────┬────STATUS─────┬PORTS┬───────NAMES────────╮
│ 0 │ 673394ef1d4c │ busybox │ "top" │ 47 seconds ago │ Up 45 seconds │     │ nostalgic_shockley │
│ 1 │ d85756f57265 │ busybox │ "top" │ 52 seconds ago │ Up 51 seconds │     │ high_albattani     │
╰───┴──────────────┴─────────┴───────┴────────────────┴───────────────┴─────┴────────────────────╯