Added WOfS errors notebook

[MatthewJA]

Proposed changes

Added a notebook that estimates WOfS errors and probabilities as part of the waterbodies toolkit - thinking of using this for waterbody edges later on.

Checklist (replace [ ] with [x] to check off)

• Notebook created using the DEA-notebooks template
• Remove any unused Python packages from Load packages
• Remove any unused/empty code cells
• Remove any guidance cells (e.g. General advice)
• Ensure that all code cells follow the PEP8 standard for code. The jupyterlab_code_formatter tool can be used to format code cells to a consistent style: select each code cell, then click Edit and then one of the Apply X Formatter options (YAPF or Black are recommended).
• Include relevant tags in the final notebook cell (refer to the DEA Tags Index, and re-use tags if possible)
• Clear all outputs, run notebook from start to finish, and save the notebook in the state where all cells have been sequentially evaluated
• Test notebook on both the NCI and DEA Sandbox (flag if not working as part of PR and ask for help to solve if needed)
• If applicable, update the Notebook currently compatible with the NCI|DEA Sandbox environment only line below the notebook title to reflect the environments the notebook is compatible with

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[robbibt]

I get an error in the "Load DEA Waterbodies" cell when trying to load the WOFS shapefile, is the idea that this bit will be replaced with one of Ness's WFS funcs later on? In the meantime it might be worth mentioning that users need access to this file in a "Prerequisites" bit up the top

---------------------------------------------------------------------------
CPLE_OpenFailedError                      Traceback (most recent call last)
fiona/_shim.pyx in fiona._shim.gdal_open_vector()

fiona/_err.pyx in fiona._err.exc_wrap_pointer()

CPLE_OpenFailedError: DigitalEarthAustraliaWaterbodies.shp: No such file or directory

During handling of the above exception, another exception occurred:

DriverError                               Traceback (most recent call last)
<ipython-input-6-8e42fa5b112a> in <module>
----> 1 wbs = gpd.read_file("DigitalEarthAustraliaWaterbodies.shp").set_index("UID")

/g/data/v10/public/modules/dea-env/20200713/lib/python3.6/site-packages/geopandas/io/file.py in _read_file(filename, bbox, mask, rows, **kwargs)
     94 
     95     with fiona_env():
---> 96         with reader(path_or_bytes, **kwargs) as features:
     97 
     98             # In a future Fiona release the crs attribute of features will

/g/data/v10/public/modules/dea-env/20200713/lib/python3.6/site-packages/fiona/env.py in wrapper(*args, **kwargs)
    396     def wrapper(*args, **kwargs):
    397         if local._env:
--> 398             return f(*args, **kwargs)
    399         else:
    400             if isinstance(args[0], str):

/g/data/v10/public/modules/dea-env/20200713/lib/python3.6/site-packages/fiona/__init__.py in open(fp, mode, driver, schema, crs, encoding, layer, vfs, enabled_drivers, crs_wkt, **kwargs)
    252         if mode in ('a', 'r'):
    253             c = Collection(path, mode, driver=driver, encoding=encoding,
--> 254                            layer=layer, enabled_drivers=enabled_drivers, **kwargs)
    255         elif mode == 'w':
    256             if schema:

/g/data/v10/public/modules/dea-env/20200713/lib/python3.6/site-packages/fiona/collection.py in __init__(self, path, mode, driver, schema, crs, encoding, layer, vsi, archive, enabled_drivers, crs_wkt, ignore_fields, ignore_geometry, **kwargs)
    152             if self.mode == 'r':
    153                 self.session = Session()
--> 154                 self.session.start(self, **kwargs)
    155             elif self.mode in ('a', 'w'):
    156                 self.session = WritingSession()

fiona/ogrext.pyx in fiona.ogrext.Session.start()

fiona/_shim.pyx in fiona._shim.gdal_open_vector()

DriverError: DigitalEarthAustraliaWaterbodies.shp: No such file or directory

[MatthewJA]

Yeah, planning to replace that. Actually I could probably do that now... Lemme do that and ping for re-review

[BexDunn]

Yeah, planning to replace that. Actually I could probably do that now... Lemme do that and ping for re-review

I'm having the same issue :) Quick comments first off would be, can you plz chuck in a quick link to a relevant ref for the marginals/ monte carlo, or give a quick one line/short para explainer ? Looks like nice code. What's Jax? I've used math jax...

[MatthewJA]

@robbibt @BexDunn Updated to use the new waterbodies code! Should be all smooth now.

[CEKrause]

This notebook looks incredible, but I am having a hard time following what you're doing. Can you please add more doco and code comments throughout?

[BexDunn]

Have looked at this version, look forward to looking again when you've squished the terrain shadow bug :)

[MatthewJA]

@CEKrause I've had a crack at better explaining the stuff that's there. Is that easier to follow? A lot of the code is in the virtual product which is mostly a bunch of boilerplate, but I've commented that too. There's also the monolithic WOfS line, but all this one is is a series of if/elses that draw out the tree, so not sure if I can add anything to that.

@BexDunn I've applied the shadow mask, which indeed masked out the problematic section. I also changed the example to one where the lake is more visible and more obviously noisy in the input image, so the shadow mask won't actually show up in the notebook as it is.

[CEKrause]

@MatthewJA I have added a few bits of doco myself and marked in caps some places where the doco could be clearer. I added the word pumpkin with all my comments so you can easily find them in the notebook and make sure you don't miss any.

[MatthewJA]

@CEKrause 🎃

[MatthewJA]

Also here's an image of the actual tree! Leaf colours are blue for wet and red for dry.

[CEKrause]

Looks awesome. Perfect changes.

[CEKrause]

Also here's an image of the actual tree! Leaf colours are blue for wet and red for dry.

Is it worth adding this to the notebook too?

[BexDunn]

Also here's an image of the actual tree! Leaf colours are blue for wet and red for dry.

Is it worth adding this to the notebook too?

I'd say yes

[BexDunn]

Looks good, could also add the tree classifier jpeg

[MatthewJA]

I might skip the image, just because the actual image from the paper is there, and this is just a worse version of that

[BexDunn]

I might skip the image, just because the actual image from the paper is there, and this is just a worse version of that

happy with that :)