Allow to customize S3 region when opening the ObjectStore

[eddyxu]

Problem Statement

Currently, lance only supports the default S3 region defined in ~/.aws/config. And our public datasets are hosted in us-west-2.

[JSpenced]

@eddyxu Are there examples of opening the object store somewhere? I want to save and load lance datasets from s3, but can't find examples anywhere at the moment.

[eddyxu]

@JSpenced in python, you should be able to

# Write
lance.write_dataset(tbl, "s3://bucket/path/dataset.lance")

# Read
dataset = lance.datasaet("s3://bucket/path/dataset.lance")

[JSpenced]

@eddyxu

I keep getting this error (normally, we infer the credentials from the IAM role of the ec2 instance. so number 9 on this page https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html, but even when I try to use a credentials file I am getting the same error):
LanceError(I/O): Generic S3 error: Error performing get request test.lance/_latest.manifest: response error "No Body", after 0 retries: HTTP status client error (403 Forbidden) for url (https://s3.us-east-1.amazonaws.com/blah/jspenced/temp/test.lance/_latest.manifest)

Also, tried doing something like this, but it only expects a string not an S3File object:

fs = s3fs.S3FileSystem()
with fs.open("jspenced/temp/test.lance", "wb") as f:
    lance.write_dataset(temp, f)

I should note that df.to_parquet("s3://path/test.parquet") works fine.

[changhiskhan]

@eddyxu

I keep getting this error (normally, we infer the credentials from the IAM role of the ec2 instance. so number 9 on this page https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html, but even when I try to use a credentials file I am getting the same error): LanceError(I/O): Generic S3 error: Error performing get request test.lance/_latest.manifest: response error "No Body", after 0 retries: HTTP status client error (403 Forbidden) for url (https://s3.us-east-1.amazonaws.com/blah/jspenced/temp/test.lance/_latest.manifest)

Also, tried doing something like this, but it only expects a string not an S3File object:

fs = s3fs.S3FileSystem()
with fs.open("jspenced/temp/test.lance", "wb") as f:
    lance.write_dataset(temp, f)

I should note that df.to_parquet("s3://path/test.parquet") works fine.

this is fixed in #775

[changhiskhan]

@eddyxu using from_env i think it should check the AWS_PROFILE variable as well

https://docs.rs/object_store/latest/object_store/aws/struct.AmazonS3Builder.html#method.from_env

[JSpenced]

Thanks for this! I will test it out and let you know if I have any issues.