Extend Iris loading to accept S3 URLs

[JoshuaWiggs]

✨ Feature Request

Add capability to the Iris loading and io modules to utilise a S3 URL to load a data cube into memory.

Proposed used would be something like
cube = iris.load("s3://some-bucket/some-object")

Motivation

This feature would allow us to make more optimal use of our AWS cloud based platforms by removing the need to copy data files from our object store to a mount file system before working with them.

Additional context

Click to expand this section...

This feature is required in order to allow us to reduce remove the duplication of the s3 input and output data. This will allow us to just have the one instance input data in the s3 input bucket being used directly by our science workflows. This will allow the reduction of our fsx storage capacity which is currently our greatest spend on AWS infrastructure.

This could be accomplished by adding a S3 loading method into Iris utilising the boto3 (https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) Python library. Therefore, this would add that library as an optional dependency to use this loading method.

The scope of this feature is limited to loading data from an S3 bucket. Additionally, this feature will assume that you have already correctly configured your environment to access the S3 bucket that is being targeted.

[ukmo-ccbunney]

From @SciTools/peloton : we should set up an internal meeting to discuss this and scope out the requirements.

[pp-mo]

OK I couldn't resist looking into this, because I thought there was a "quick win".
On reviewing the features of netcdf4-python, I think it does after all provide what is needed : it can create an in-memory Dataset from a byte buffer

I have made a POC here

In short, this shows that you can load a netcdf4 file in a memory buffer, make a netCDF4.Dataset from that, and load that direct into Iris.
I presume that S3 loading can generate a bytes buffer object, just like a file load. Don't actually know too much about that bit.

[pp-mo]

S3 loading can generate a bytes buffer object

maybe relevant ? https://stackoverflow.com/questions/36205481/read-file-content-from-s3-bucket-with-boto3

[pp-mo]

Just demo-ed this at AVD "Surgery" day
Very exciting to see it working!

But then ... found that Netcdf "Proxy" objects don't function on in-memory datasets because of expectation to re-open from a dataset path.
Temporarily fixed by iris.fileformats.netcdf.loader._LAZYVAR_MIN_BYTES = 1e10
This could probably be fixed by adding awareness of in-memory datasets to our creation of the DataProxy objects. Or possibly, it makes sense to just "not do lazy" in these cases, as above enforces ??

Since then, also, discussed whether supporting S3 as a specific url option is a "Good Idea" ™

• ✔ utility for users : just replace path with url
• ❌ means extra dependency (boto3?) and feature maintenance
• ❌ most existing filetypes are plain unsuitable, since S3 doesn't allow partial access
  • .... and we don't (yet) support Zarr (directly), which is the obvious choice
  • ?? but could re-introduce a kind of laziness
    ==> scan one-by-one (multiple smaller files -- see below) ; extract metadata; don't store it all
• probably need to support wildcards + multiples, as for regular loads
• probably support other filetypes, e.g. PP --> ?? all, i.e. (somehow) independent of type ??

[JoshuaWiggs]

Script used to demonstrate loading from an S3 URL

import boto3
import iris
import iris.fileformats.netcdf.loader
import netCDF4 as nc
import iris.quickplot as qplt
import matplotlib.pyplot as plt


def main():

    # Create an S3 client
    s3 = boto3.client('s3')

    # Specify the bucket name and file name
    bucket_name = 's3_bucket'
    file_name = 'file.nc'

    # Upload the file to S3
    obj = s3.get_object(
        Bucket=bucket_name,
        Key=file_name
    )

    # Read the file content
    file_content = obj['Body'].read()

    ncds = nc.Dataset("in-memory.nc", memory=file_content)

    iris.fileformats.netcdf.loader._LAZYVAR_MIN_BYTES = 1e12
    cubes = iris.load(ncds)

    for cube in cubes:
        qplt.contourf(cube)
        plt.show()
    

if __name__ == "__main__":
    main()

[JoshuaWiggs]

@cpelley pinging you for your interest

[trexfeathers]

Let's go for it 👍

We'd like to deliver this in Iris 3.14. To use our typical sprint sizing, we expect this to be a 'large' task.

Concerns we had

• Building in Boto3 knowledge into Iris, so it becomes part of our maintenance burden?
  • Boto3 apparently has a reputation for stability, and uses SemVer.
  • All our code would be in a small corner of Iris, so wouldn't get in many people's way when developing.
• Are we encouraging bad practices? I.e. NetCDF on S3 ONLY enables access to the whole file, never part of it.
  • The front-end API we write could extend to PP/GRIB (or Zarr in future), all of which make a lot more sense for this.
  • With NetCDF, there is no alternative that we would be discouraging.

Design

• Should be an experimental feature.
  • We'll be relying on users to try it out.
  • Can achieve this by adding a run-time flag in iris.experimental, which can be checked in an if: block during the standard loading process. Previous example:
    

    iris/lib/iris/experimental/ugrid.py

    Lines 135 to 136 in e4191fb

    	#: Run-time switch for experimental UGRID-aware NetCDF loading. See :class:`~iris.mesh.load.ParseUGridOnLoad`.
    	PARSE_UGRID_ON_LOAD = ParseUGridOnLoad()

• The loading 'chain' should automatically detect S3 URL(s) in the URI(s) provided.
• Design should allow for future S3 loading from other file formats.
  • PP and GRIB both use self-describing 2D slices ('fields'), so are ideal for downloading subsets of a larger dataset.
  • Zarr is designed with cloud in mind, although I don't know the details yet.
• Should only support NetCDF for now - using the approach described in previous comments here.
• Laziness should be disabled for S3 loading - we can't see an alternative when loading directly from memory.
• Attempt to support wildcards in the URI(s), if possible.

[JoshuaWiggs]

This is excellent news, let me know if there is anything we can do to help you with this process.

I was going to reach out about the possibility of adding a save to S3 to complete the I/O offering from Iris. I haven't investigated the feasibility of this integration yet, but I'm guessing it would be best to raise as a separate issue?

[pp-mo]

@JoshuaWiggs save to S3

Not thought of that case !
I think probably can be done diskless, somehow, using the same 'in-memory-dataset' option in netCDF4-python, as in the above example.

As it was for loading, Iris can already save direct to an open netcdf dataset ( #5214 ), as this was likewise needed for ncdata integration.

However the catch is, I think, you must specify the netcdf loader function specifically as because it isn't a filepath with an extension it can't choose automatically.
The ncdata "from_iris" function uses it here :
In fact, as you can see, the necessary code to do this is perhaps unexpectedly "fussy".

---

@JoshuaWiggs feasibility of this integration ... would be best to raise as a separate issue?

@trexfeathers do we want to consider this for inclusion here -- or is that unwanted scope creep at this point ??

[ESadek-MO]

from @SciTools/peloton:

We think implementing saving is a good idea, but we do think it should be raised as a separate issue to avoid scope creep. Thanks @JoshuaWiggs

[JoshuaWiggs]

New issue #6535 raised for saving.