Release/3.0.0

[mpu-creare]

• Bugfixes for Release 3.0.0

Need to test/fix all the data sources:

• SMAP_EGI
• weathercitizen
• GFS
• Satutils (landsat & Sentinel2)
• soilscape

[coveralls]

Coverage decreased (-0.3%) to 92.195% when pulling 70edbc0 on release/3.0.0 into 015d0eb on develop.

[coveralls]

Coverage increased (+0.1%) to 92.652% when pulling 7c810e1 on release/3.0.0 into 015d0eb on develop.

[jmilloy]

@mpu-creare We have the pairs of datasource nodes, e.g. RasterioBase and Rasterio. I thought that was for backwards compatibility, but with the bump to 3.0, we wouldn't need to do that. Can we get rid of the InterpolationMixin and just require the Interpolate node when you want use it?

[mpu-creare]

I'll poll the group about that at the meeting tomorrow.

[jmilloy]

@mpu-creare FYI, currently working on removing the InterpolationMixin.

[jmilloy]

@mpu-creare Does the algorithm Reproject class replace ReprojectedSource? Should we deprecate ReprojectedSource for this release?

[jmilloy]

Same, 1.1.8.

I have

AWS_ACCESS_KEY_ID <XXXX>
AWS_SECRET_ACCESS_KEY <XXXX>
AWS_REGION_NAME us-east-1
AWS_TAGS None
AWS_BUDGET_AMOUNT None
AWS_BUDGET_EMAIL None
AWS_REQUESTER_PAYS False
S3_BUCKET_NAME None

[jmilloy]

boto3 1.16.30

[mpu-creare]

Do you have boto3 configured in your environment? I don't.

[jmilloy]

How do I find out if I have boto3 configured?

[jmilloy]

I don't have any AWS environmental variables defined, but I do have a ~/.aws/config file. It just has:

[default]
region_name=us-east-1

[user jxm@creare]
region_name=us-east-1

[mpu-creare]

boto3.__version__ '1.17.43'

in a fresh session, try:

import boto3
s = boto3.Sesssion()
s.get_credentials()

I have to switch tasks for now... sorry...

[mpu-creare]

looks like boto3 is NOT configured... hhmmmm... Right now TestGFS.test_source passes for me with yesterday = now - datetime.timedelta(days=24)

[jmilloy]

Yeah, my boto3 session credentials are populated with a access_key and secret_key

[jmilloy]

That'll be from ~/.aws/credentials.

[jmilloy]

Removing the .aws/credentials did remove the boto3 session credentials, but it didn't fix the error.

[mpu-creare]

Hmmm. these are my settings.

{
 'DEBUG': False,
'MULTITHREADING': False,
 'DEFAULT_CACHE': [],
 'CACHE_DATASOURCE_OUTPUT_DEFAULT': True,
 'CACHE_NODE_OUTPUT_DEFAULT': False,
 'AWS_ACCESS_KEY_ID': 'xxxxxxxxxxxxx',
 'AWS_SECRET_ACCESS_KEY': 'xxxxxxxxxxxx',
 'AWS_REGION_NAME': 'US-EAST-1',
 'AWS_REQUESTER_PAYS': False,
}

I'm also have the following in the test, and this works too:

        with podpac.settings:
            podpac.settings['AWS_ACCESS_KEY_ID'] = None
            podpac.settings['AWS_SECRET_ACCESS_KEY'] = None
            podpac.settings['AWS_REGION_NAME'] = None
            ...

[mpu-creare]

That RasterioIOError that you get -- is that from a fresh try? or after retrying.

I find that often rasterio throws and error, and then on a second pass throws THIS particular not recognized as a supported file format error.

[jmilloy]

>>> import rasterio
>>> session = rasterio.session.AWSSession(
        region_name='us-east-1',
...     aws_unsigned=True)
>>> with rasterio.env.Env(session=session) as env:
...     dataset = rasterio.open('s3://noaa-gfs-pds/SOIM/0-10 m DPTH/20210315/1200/003')
... 
Traceback (most recent call last):
  File "rasterio/_base.pyx", line 216, in rasterio._base.DatasetBase.__init__
  File "rasterio/_shim.pyx", line 67, in rasterio._shim.open_dataset
  File "rasterio/_err.pyx", line 213, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_OpenFailedError: '/vsis3/noaa-gfs-pds/SOIM/0-10 m DPTH/20210315/1200/003' not recognized as a supported file format.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/home/jmilloy/Creare/Pipeline/_podpac-38_/lib/python3.8/site-packages/rasterio/env.py", line 433, in wrapper
    return f(*args, **kwds)
  File "/home/jmilloy/Creare/Pipeline/_podpac-38_/lib/python3.8/site-packages/rasterio/__init__.py", line 221, in open
    s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
  File "rasterio/_base.pyx", line 218, in rasterio._base.DatasetBase.__init__
rasterio.errors.RasterioIOError: '/vsis3/noaa-gfs-pds/SOIM/0-10 m DPTH/20210315/1200/003' not recognized as a supported file format.

[mpu-creare]

I just ran your code above from a fresh session with no issues.

Could you try the following for me?

with rasterio.env.Env(session=session, AWS_HTTPS=False) as env:
     dataset = rasterio.open('s3://noaa-gfs-pds/SOIM/0-10 m DPTH/20210315/1200/003')

[jmilloy]

Yeah, that works! So...?

[mpu-creare]

I think it means there's some issue with ssl certificates... rasterio cannot find them for some reason on your machine. I've been debugging a similar issue on the weatherground server.... it's a bit mysterious to me.

You can try setting:

export 'CURL_CA_BUNDLE'='/etc/ssl/certs/ca-certificates.crt'

if you're on linux... and let me know if that works WITHOUT the AWS_HTTPS=False

Make sure it's set properly by checking os.environ inside Python.

[jmilloy]

Yeah, setting CURL_CA_BUNDLE worked.

[jmilloy]

Here's a commit that "fixes" this issue by documenting it rasterio/rasterio@b621d92

rasterio or curl or something should report a more informative error. Do you want me to try to do that in podpac, at least?