Unable to register tables in non-cloud S3 servers

[avantgardnerio]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

In DataFusion, it appears there are S3 implementations for common cloud providers: AWS, Azure, GCP, etc. These implementations assume cloud-provider-specific things like REGION and default port. When trying to register a table at an arbitrary address in minio, it fails:

/snap/bin/cargo run --color=always --bin datafusion-cli --manifest-path /home/bgardner/workspace/ballista/arrow-datafusion/datafusion-cli/Cargo.toml
    Finished dev [unoptimized + debuginfo] target(s) in 0.09s
     Running `target/debug/datafusion-cli`
DataFusion CLI v12.0.0
create external table customer stored as CSV DELIMITER '|' location 's3://minioadmin:minioadmin@127.0.0.1:9000/tpch/customer.tbl';
Internal("No suitable object store found for s3://minioadmin:minioadmin@127.0.0.1:9000/tpch/customer.tbl")

Describe the solution you'd like

The above syntax work.

Describe alternatives you've considered

N/A

[avantgardnerio]

Note: it looks like it may be possible with the correct environment variables configured. If so, then maybe this issue is just a documentation request.

[avantgardnerio]

It appears from this PR that the bucket name needs to be compiled into Rust. I'll file another issue for making this dynamic at runtime, which is what will be required for our use-case / Ballista support.

[avantgardnerio]

CC @andygrove

[avantgardnerio]

I can confirm AWS_ENDPOINT works. Closing this - I have no idea how it stayed open so long.