Add S3 to Browser panel

[andrewharvey]

Feature description.
The Browser panel is neat, I can list a local directory, list all my PostGIS databases and tables, etc. it would be nice to have an S3 item here where you could add buckets, then expanding each bucket would list all the files in the bucket just like the file directory.

This makes it a lot easier and faster to work with data stored in S3 buckets without needing to manually enter the bucket name and object key for each file in the add vector/raster dialog.

A workaround is to mount the bucket on your local file system using something like https://github.com/s3fs-fuse/s3fs-fuse, but that takes configuration and setup, so something out of the box with QGIS would be nice.

[palmerj]

Yes would be a great feature. How would you see this feature working with access keys?

[andrewharvey]

Yes would be a great feature. How would you see this feature working with access keys?

With QGIS 3.12 you can define any of the GDAL AWS variables (https://gdal.org/user/virtual_file_systems.html#vsis3-aws-s3-files-random-reading) under Project | Variables which is very nice and what I do (since sometimes I use other services which are S3 compatible so I need to set the endpoint variable anyway).

I'm also assuming these variables are set if exported in the shell which you start qgis (though not many people would start qgis from the shell).

I guess though you'd want something similar to what it does for PostgreSQL, you choose a new bucket (instead of new connection for psql) and set the keys in a UI dialog.

[NathanW2]

That is a neat idea and would be quite easy to implement I suspect. For access keys we could use the password store that QGIS has already to manage them I suspect right?

…

On Mon, Mar 30, 2020 at 3:43 PM Jeremy Palmer ***@***.***> wrote: Yes would be a great feature. How would you see this feature working with access keys? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#35450 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC5FXFE27AD444C4HWRSMTRKAWQ7ANCNFSM4LWHIYWA> .

[palmerj]

I'm also assuming these variables are set if exported in the shell which you start qgis (though not many people would start qgis from the shell).

I set the GDAL environment variables in QGIS so they override the system environment variables. But either way they are QGIS instance wide and I use S3 buckets that require different keys because they come from different AWS accounts.

I guess though you'd want something similar to what it does for PostgreSQL, you choose a new bucket (instead of new connection for psql) and set the keys in a UI dialog.

Yes and these keys only apply to that bucket connection. As @NathanW2 mentions using the QGIS password store would be good.

[savmickael]

👍

[lobenichou]

Just wanted to chime in here that it would be quite useful to be able to add s3 as a favorite in the browser panel, especially when using COG files. This is a bit old so I wanted to revive the thread.

[drwelby]

+1 to the above: with cloud-optimized file formats like STAC and COG becoming more popular this would be a very useful feature.

[rockliffelewis]

Yes would be a great feature. How would you see this feature working with access keys?

It would be nice to be able to specify multiple end points with different credentials for each to use as a root in the browser tree which is not very easy to do with the GDAL VSI environment variables.

It would also be good to be able to specify a prefix to use as the root of the browser tree for occasions where the access given is limited to a prefix rather than the root of the bucket

Cloud based delivery of Earth Observation raster imagery is becoming more prevalent, and this would enable some nice usage patterns within QGIS

[nyalldawson]

I'm working on this for 3.40 👍

[rockliffelewis]

Yes would be a great feature. How would you see this feature working with access keys?

It would be nice to be able to specify multiple end points with different credentials for each to use as a root in the browser tree which is not very easy to do with the GDAL VSI environment variables.

I figured out a way to do this using the GDAL_CONFIG environment variable and creating a config file with the various s3 endpoints and credentials like this https://gdal.org/user/virtual_file_systems.html#how-to-set-credentials

It would be great if this could be integrated into the QGIS key store though

[rockliffelewis]

I also attempted to deal with adding s3 to the qgis key store here:
#53778

which might be useful to you, although im not sure my approach was the best one

[nyalldawson]

@rockliffelewis
I did see that! My current project won't involve the authentication framework at all (rather I am targeting the plain text storage of the connection options), but it WILL make it much easier for an authentication framework plugin (like yours) to be completed

[robbibt]

Hey @nyalldawson, have been trying to work out whether this feature is already available in a current version of QGIS, or is it due for release down the track?

[agiudiceandrea]

@robbibt, it will be available since QGIS 3.40. It is currently available using a recently built QGIS 3.39 developing version.