reading from stdin in datafusion-cli?

[13minutes-yt]

Am I missing something or datafusion-cli has no support for reading csv from stdin yet?
duckdb has it like this example
cat taxi.csv|duckdb -c "select * from read_csv('/dev/stdin')"

[SteveLauC]

I tried this with JSON:

$ cat sql
create external table test stored as json location '/dev/stdin';
select * from test;

$ cat 1.json
{ "age": 2, "name": "steve" }

$ cat 1.json |datafusion-cli -f sql
DataFusion CLI v36.0.0
0 rows in set. Query took 0.001 seconds.

0 rows in set. Query took 0.000 seconds.

It won't give you error, though it won't print the rows as well..

---

For parquet, datafusion-cli gives you an error because the file size of a pipe is 0:

$ cat src/main.rs
fn main() {
    let stat = nix::sys::stat::fstat(0).unwrap();
    println!("{}", stat.st_size);
}

$ cargo b
$ cat 1.json | ./target/debug/rust
0

$ cat sql
create external table test stored as parquet location '/dev/stdin';
select * from test;

$ pqrs cat 2.parquet

###############
File: 2.parquet
###############

{Age: 2}

$ cat 2.parquet | datafusion-cli -f sql
DataFusion CLI v36.0.0
Execution error: file size of 0 is less than footer
Error during planning: table 'datafusion.public.test' not found

---

For CSV, it gives you an error Object Store error: Generic LocalFileSystem error: Error seeking file /dev/stdin: Illegal seek (os error 29) because pipe is not seekable:

$ cat sql
create external table test stored as csv location '/dev/stdin';
select * from test;

$ cat username.csv
Username; Identifier;First name;Last name
booker12;9012;Rachel;Booker
grey07;2070;Laura;Grey
johnson81;4081;Craig;Johnson
jenkins46;9346;Mary;Jenkins
smith79;5079;Jamie;Smith


$ cat username.csv | datafusion-cli -f sql
DataFusion CLI v36.0.0
Object Store error: Generic LocalFileSystem error: Error seeking file /dev/stdin: Illegal seek (os error 29)
Error during planning: table 'datafusion.public.test' not found

[alamb]

I don't think you are missing anything -- it does not have support for reading from stdin. I think it would be a good feature.

I bet if you filed a ticket someone would try and implement it .

[alamb]

The issue is tracked by #9409 now -- thank you @13minutes-yt

[lij55]

besides datafusion-cli, it could be much better if /dev/stdin could work with register_csv. The input is from stdin in some special cases. Thanks!