How to register a http url to the object_store

[elliot14A]

I was trying to register a http url to the object_store I was not able to find any examples on how to do it. I was able to successfully use and register s3 urls but, cannot find a way on how to use http.
register_object takes scheme, host, object_store as arguments can anybody help on how to pass http url to this function and register.

[tustvold]

What API does this HTTP server expose? Is it an object store?

[elliot14A]

Yes @tustvold !

[tustvold]

Depending on the API it exposes you should be able to do something like

let store = AmazonS3Builder::new()
    .with_bucket_name(...)
    .with_endpoint(...)
    .build()
    .unwrap();

ctx.runtime_env()
        .register_object_store("http", "", store);

[elliot14A]

For example if I have a http url https://some_example.com/blogs.csv

How should I use it to build AmazonS3Builder
what should I pass to with_bucket_name & with_endpoint as parameters

[tustvold]

Currently DataFusion only supports querying from remote object stores, not arbitrary HTTP servers. I've created apache/arrow-rs#3294 to track adding limited support for this, but DataFusion may require list support, which isn't possible on an arbitrary HTTP server AFAIK

[tustvold]

This was added in apache/arrow-rs#3380 and will be included in the next release apache/arrow-rs#3422

[tustvold]

This has been released in object_store 0.5.3 - https://docs.rs/object_store/latest/object_store/http/struct.HttpBuilder.html