Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support remote datasets from s3 compliant data sources #7453

Merged
merged 5 commits into from
Nov 29, 2023
Merged

Conversation

frcroth
Copy link
Member

@frcroth frcroth commented Nov 22, 2023

URL of deployed dev instance (used for testing):

  • https://___.webknossos.xyz

Steps to test:

  • Set up a minio server
  • Create a bucket in that server and upload a dataset to it (do this via the web console)
  • Create an access key on the server
  • Explore this dataset with URL: "s3://<minio-server-ip>:<minio-server-port>/<bucket-name>/<root-key-of-the-dataset>"
  • View dataset
  • Also explore S3 datasets to ensure they still work
Alternative repl testing
import java.net.URI;
import com.scalableminds.webknossos.datastore.datavault.S3DataVault;
implicit val ec: scala.concurrent.ExecutionContext = scala.concurrent.ExecutionContext.global;
import com.scalableminds.webknossos.datastore.datavault.VaultPath;
val key = com.scalableminds.webknossos.datastore.storage.S3AccessKeyCredential(name="someKey", accessKeyId="FhXRPMiC9KWMBh9WXF54", secretAccessKey="9eVWQYZckmshjIMsebOBkztMqfHpC5t88bFWOMhc", user="", organization="");
val uri =  new URI("s3://10.191.1.63:8000/somezarr/e75_zarr_local");
val minioVault = new S3DataVault(Some(key), new URI("s3://10.191.1.63:8000/somezarr/e75_zarr_local"));
val vp = new VaultPath(uri, minioVault);
(vp / "color" / "1" / "credentials.json").readBytes().get("")

TODOs:

  • ...

Issues:


(Please delete unneeded items, merge only when none are left open)

@frcroth frcroth requested a review from fm3 November 22, 2023 13:31
@frcroth frcroth self-assigned this Nov 27, 2023
Copy link
Member

@fm3 fm3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good :) Added a comment on code style. Feel free to merge after tackling that

@frcroth frcroth merged commit 4487180 into master Nov 29, 2023
2 checks passed
@frcroth frcroth deleted the s3-compliant branch November 29, 2023 08:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support S3-compliant storages
2 participants