[REVIEW] Sea ice forecasting using the IceNet library

[acocac]

Notebook Review: Issue #2

Submitting author: @bnubald

Repository: https://github.com/eds-book-gallery/67a1e320-7c47-4ea9-8df8-e868326bc90b

Notebook idea issue: eds-book/notebooks-ideas#5

Editor: @annefou

Reviewer: @weiji14 @William-gregory

Managing EiC: @acocac

Status

Reviewer instructions & questions

Hi 👋 @weiji14 & @William-gregory, please carry out your review in this issue by updating the checklist below.

As a reviewer, you contribute to the technical quality of the content published by our community.

Before the review, EiC checked if the submission fits the minimum requirements.

The quality of the proposed contribution can be assessed through scientific, technical and code criteria.

The reviewer guidelines are available here: https://edsbook.org/publishing/guidelines/guidelines-reviewers.html.
Any questions/concerns please let @annefou know.

Review checklist for @weiji14

Please check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide.

Conflict of interest

• As the reviewer I confirm that there are no conflicts of interest for me to review this work (If you are unsure whether you are in conflict, please speak to your editor before starting your review).

Code of conduct an peer-review principles

• I confirm that I read and will adhere to the Environmental Data Science code of conduct.
• I confirm that I read and will adhere to Open Science Peer Review Oath

General checks

• Notebook: Is the notebook file (notebook.ipynb) part of the notebook repository?
• Contribution and authorship: Does the author list seem appropriate and complete (full name, affiliation, and GitHub/ORCID handle)?
• Scope and eligibility: Does the submission contain an original and complete analysis according to the scope of EDS book?

Reproducibility

• Does the notebook run in a local environment?
• Does the notebook build and run in binder?
• Are all data sources openly accessible and properly cited (e.g. with citation to a persistent DOI) in the heading section?

Pedagogy

• Are the notebook purpose and highlights clear?
• Does the notebook demonstrate some specific data analysis or visualisation techniques?
• Is the notebook well documented, using both markdown cells and comments in code cells?
• Does the conclusion section provide clear and concise final say on the tools, analysis and/or datasets used?
• Is the notebook narrative well written (it does not require editing for structure, language, or writing quality)?

Ethical

• Is any linkage of datasets in the notebook unlikely to lead to an increased risk of the personal identification of individuals?
• Is the notebook truthful and clear about any limitations of the analysis (and potential biases in data and/or tools)?
• Is the notebook unlikely to lead to negative social outcomes, such as (but not limited to) increasing discrimination or injustice?

Other Requirements

• All mentioned software should be formally and consistently cited wherever possible.
• Acronyms should be spelled out upon first mention.
• License conditions on images and figures must be respected (Creative Commons, etc.).

Final approval (post-review)

• Authors has responded to my review and made changes to my satisfaction. I recommend approving the notebook for publication.

Review checklist for @William-gregory

Please check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide.

Conflict of interest

• As the reviewer I confirm that there are no conflicts of interest for me to review this work (If you are unsure whether you are in conflict, please speak to your editor before starting your review).

Code of conduct an peer-review principles

• I confirm that I read and will adhere to the Environmental Data Science code of conduct.
• I confirm that I read and will adhere to Open Science Peer Review Oath

General checks

• Notebook: Is the notebook file (notebook.ipynb) part of the notebook repository?
• Contribution and authorship: Does the author list seem appropriate and complete (full name, affiliation, and GitHub/ORCID handle)?
• Scope and eligibility: Does the submission contain an original and complete analysis according to the scope of EDS book?

Reproducibility

• Does the notebook run in a local environment?
• Does the notebook build and run in binder?
• Are all data sources openly accessible and properly cited (e.g. with citation to a persistent DOI) in the heading section?

Pedagogy

• Are the notebook purpose and highlights clear?
• Does the notebook demonstrate some specific data analysis or visualisation techniques?
• Is the notebook well documented, using both markdown cells and comments in code cells?
• Does the conclusion section provide clear and concise final say on the tools, analysis and/or datasets used?
• Is the notebook narrative well written (it does not require editing for structure, language, or writing quality)?

Ethical

• Is any linkage of datasets in the notebook unlikely to lead to an increased risk of the personal identification of individuals?
• Is the notebook truthful and clear about any limitations of the analysis (and potential biases in data and/or tools)?
• Is the notebook unlikely to lead to negative social outcomes, such as (but not limited to) increasing discrimination or injustice?

Other Requirements

• All mentioned software should be formally and consistently cited wherever possible.
• Acronyms should be spelled out upon first mention.
• License conditions on images and figures must be respected (Creative Commons, etc.).

Final approval (post-review)

• Authors has responded to my review and made changes to my satisfaction. I recommend approving the notebook for publication.

Additional instructions

Reviewer general comments are welcome on this REVIEW issue or directly to the notebook repository.

If you do the latter, you will find a Pull Request titled REVIEW where you can carry out the discussion with authors through ReviewNB, a third-party plugin in GitHub for displaying and commenting Jupyter Notebooks (see further details here).

In addition to ReviewNB, we suggest to explore or run the notebook in:

• Binder (run): Click the Launch Binder button at the top level of this message.

[annefou]

👋 @weiji14 @William-gregory we will conduct the review in this issue.
Please read through the above information and let me know if you have any questions about the review process.
Thank you 🙏

[annefou]

Hi @weiji14 @William-gregory I am just checking in on the status of the review. Are there any obstacles or is there anything I can assist you with?

[William-gregory]

Hi @annefou, apologies for the delay. I've just returned from vacation and will be getting on with the review this week. One initial point of note is that my environment build fails on my local machine (works fine on Binder). Am I doing something wrong?

Reproducibility

• Does the notebook run in a local environment?

Building the environment fails:

git clone https://github.com/eds-book-gallery/67a1e320-7c47-4ea9-8df8-e868326bc90b.git
cd 67a1e320-7c47-4ea9-8df8-e868326bc90b
conda env create -f .binder/environment.yml -n IceNet

ERROR: Failed building wheel for netCDF4
ERROR: Could not build wheels for netCDF4, which is required to install pyproject.toml-based projects

• Does the notebook build and run in binder? (mybinder repo)

[bnubald]

Hi @William-gregory, could I ask if you're using Windows?

The library is supported on Linux (potentially Unix).

[William-gregory]

Hi @bnubald, I'm attempting to run on MacOS

[annefou]

On my side, I tried in mybinder.org and on my laptop (creating a local environment as you did).

It works in mybinder.org out of the box but not on my laptop (MacOS too). But then I changed netCDF version to 1.6.5 and it seems to work. So maybe the version of netCDF could be changed to 1.6.5?

[bnubald]

There could be a potential issue with versions > 1.6.0 (icenet-ai/icenet#226). But, if its working, happy to consider.

Else, maybe using conda for the netcdf4 install which has macos listed as supported.

name: 67a1e320-7c47-4ea9-8df8-e868326bc90b
channels:
  - conda-forge
  - defaults
dependencies:
  - libnetcdf
  - pip
  - python<3.12
  - "tensorflow<2.16=cpu*"
  - netcdf4=1.6.0
  - pip:
      - icenet @ git+https://github.com/icenet-ai/icenet@3654dc4954eca6d28e16b4876bd6538abd1f0c06
      - ecmwflibs

[William-gregory]

I managed to get the install working by using netCDF4 version 1.6.5. I also included jupyter in the dependencies in order to launch the notebook. I'm now getting an error on from icenet.data.interfaces.cds import ERA5Downloader

ImportError: dlopen(/opt/miniconda3/envs/IceNet/lib/python3.11/site-packages/cf_units/_udunits2.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_ut_ignore'

[bnubald]

I managed to get the install working by using netCDF4 version 1.6.5. I also included jupyter in the dependencies in order to launch the notebook. I'm now getting an error on from icenet.data.interfaces.cds import ERA5Downloader

ImportError: dlopen(/opt/miniconda3/envs/IceNet/lib/python3.11/site-packages/cf_units/_udunits2.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_ut_ignore'

Would this happen to be on an Apple Silicon system?

Could you try using this as your conda env file please?

name: 67a1e320-7c47-4ea9-8df8-e868326bc90b
name: icenet-edsbook
channels:
  - conda-forge
  - defaults
dependencies:
  - cf-units
  - libnetcdf
  - netcdf4=1.6.0
  - pip
  - python<3.12
  - "tensorflow<2.16=cpu*"
  - pip:
      - ecmwflibs
      - icenet @ git+https://github.com/icenet-ai/icenet@3654dc4954eca6d28e16b4876bd6538abd1f0c06
      - notebook

[William-gregory]

Thanks @bnubald, that's done the trick. The notebook runs through fine now

[weiji14]

Review checklist for @weiji14

Please check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide.

Conflict of interest

• As the reviewer I confirm that there are no conflicts of interest for me to review this work (If you are unsure whether you are in conflict, please speak to your editor before starting your review).

Code of conduct an peer-review principles

• I confirm that I read and will adhere to the Environmental Data Science code of conduct.
• I confirm that I read and will adhere to Open Science Peer Review Oath

General checks

• Notebook: Is the notebook file (notebook.ipynb) part of the notebook repository?
• Contribution and authorship: Does the author list seem appropriate and complete (full name, affiliation, and GitHub/ORCID handle)?

  Author names, affiliation and GitHub handle sighted in notebook.ipynb, but could not find ORCID. I'd highly recommend ORCID as a persistent identifier, and would also suggest placing all of these information in a CITATION.cff (https://citation-file-format.github.io) file if possible.

• Scope and eligibility: Does the submission contain an original and complete analysis according to the scope of EDS book?

Reproducibility

• Does the notebook run in a local environment?

  Installation worked locally on my Linux machine, and I was able to run all the notebook cells from start to end.

  However, I tried to solve for the conda environment (at https://github.com/eds-book-gallery/67a1e320-7c47-4ea9-8df8-e868326bc90b/blob/main/.binder/environment.yml) on other operating systems using conda-lock. OSX-64 (Intel chip) and OSX-ARM64 (M1/M2/M3 chips) works ok too, but I got the following errors for Windows:

  $ conda-lock lock --mamba --file environment.yml --platform win-64
  Locking dependencies for ['linux-64', 'osx-64', 'osx-arm64', 'win-64']...
  INFO:conda_lock.conda_solver:linux-64 using specs ['libnetcdf', 'pip *', 'python <3.12', 'tensorflow <2.16 cpu*']
  Failed to parse json, Expecting value: line 1 column 1 (char 0)
  Could not lock the environment for platform win-64
  Could not solve for environment specs
  The following package could not be installed
  └─ tensorflow <2.16 cpu* does not exist (perhaps a typo or a missing channel).
  {
      "success": false
  }
  

  Recommendations:

  • State what operating systems / platforms this notebook is supported for, or fix installation issues on Windows. Will need to ensure that command-line interfaces work also on Windows if choosing the latter.
  • Mention that this notebook should be able to run on a CPU device, and that a GPU won't be needed. Might need to test if this runs on macOS ARM chips.
  • Conda environment.yml file does not have an explicit dependency on jupyterlab, need to add this to be able to launch Jupyter Lab directly after local installation.
  • Recommend to not have both conda-forge and defaults channels in the conda environment.yml file (see ). Use nodefaults instead to explicitly disable channel mixing, see https://conda-forge.org/docs/user/tipsandtricks/#using-multiple-channels for context.

• Does the notebook build and run in binder?

  Yes, tested that Binder build at https://mybinder.org/v2/gh/eds-book-gallery/67a1e320-7c47-4ea9-8df8-e868326bc90b/review works, and that notebooks runs to the end (even on 4GB of RAM)!

• Are all data sources openly accessible and properly cited (e.g. with citation to a persistent DOI) in the heading section?

  ERA5, ORA5 and OSI SAF's sea-ice concentration data sources are linked to, but not properly cited with a DOI.

Pedagogy

• Are the notebook purpose and highlights clear?

  Purpose and highlights are clearly articulated at the start of the notebook.

• Does the notebook demonstrate some specific data analysis or visualisation techniques?

  The notebook shows an end-to-end example of how to download climate and sea-ice concentration data, pre-process it into a machine-learning ready format (TFRecord), train a U-Net based model on the data, and then visualise the results as an animation of predicted sea-ice concentration over time. Very nicely done!

• Is the notebook well documented, using both markdown cells and comments in code cells?

  Yes, plenty of markdown documentation in between code cells. Did leave some recommendations at REVIEW 67a1e320-7c47-4ea9-8df8-e868326bc90b#5 (comment) though on adding some extra code comments.

• Does the conclusion section provide clear and concise final say on the tools, analysis and/or datasets used?
• Is the notebook narrative well written (it does not require editing for structure, language, or writing quality)?

  Mostly ok, see minor suggestions at REVIEW 67a1e320-7c47-4ea9-8df8-e868326bc90b#5

Ethical

• Is any linkage of datasets in the notebook unlikely to lead to an increased risk of the personal identification of individuals?
• Is the notebook truthful and clear about any limitations of the analysis (and potential biases in data and/or tools)?

  On the analysis, the limitations of the neural network method are not explicitly stated, but they do mention that IceNet is a probabilistic model, rather than a physics-informed model. The authors mention that an ensemble of trained models is able to generate forecasts with a lead time of 6 months, though these metrics are not quantified against the groundtruth in the notebook. Since a train/val/test set is generated, maybe it would be good to report the test RMSE values to give an indication of performance. Also, there is no mention of how good the model is at predicting extreme highs/lows in sea-ice concentration, only mean/averages, but that is a whole other topic for discussion.

  On the datasets, there is some mention of how ERA5 and ORA5 are reanalysis datasets which have no spatiotemporal gaps, but no mention about the uncertainties associated with these datasets that are derived from physical and observational data. Are they biased to certain geographies (e.g. the Northern Hemisphere has more historical data coverage) and/or climate regimes? Does the model only work on contemporary timescales for the ERA5 period (1940-present), or can it be used for paleo-timescales (spanning thousands of years), etc?

• Is the notebook unlikely to lead to negative social outcomes, such as (but not limited to) increasing discrimination or injustice?

  There might be some geopolitical implications with decreasing sea-ice, especially in the Arctic, but unsure if this needs to be mentioned in the notebook.

Other Requirements

• All mentioned software should be formally and consistently cited wherever possible.

  Missing citations for cartopy, matplotlib, xarray, etc.

• Acronyms should be spelled out upon first mention.

  See review comments at REVIEW 67a1e320-7c47-4ea9-8df8-e868326bc90b#5.

• License conditions on images and figures must be respected (Creative Commons, etc.).

[William-gregory]

Conflict of interest

• As the reviewer I confirm that there are no conflicts of interest for me to review this work (If you are unsure whether you are in conflict, please speak to your editor before starting your review).

Code of conduct an peer-review principles

• I confirm that I read and will adhere to the Environmental Data Science code of conduct.
• I confirm that I read and will adhere to Open Science Peer Review Oath

General checks

• Notebook: Is the notebook file part of the PR?

• Contribution and authorship: Does the author list seem appropriate and complete (full name, affiliation, and GitHub/ORCID handle)?

  No ORCID, but names, affiliations and GitHub handles present

• Scope and eligibility: Does the submission contain an original and complete analysis according to the theme selected?

Reproducibility

• Does the notebook run in a local environment?

  Runs on local MacOS (silicon M1 chip) only with new environment file:

  name: 67a1e320-7c47-4ea9-8df8-e868326bc90b
  name: icenet-edsbook
  channels:
    - conda-forge
    - defaults
  dependencies:
    - cf-units
    - libnetcdf
    - netcdf4=1.6.0
    - pip
    - python<3.12
    - "tensorflow<2.16=cpu*"
    - pip:
        - ecmwflibs
        - icenet @ git+https://github.com/icenet-ai/icenet@3654dc4954eca6d28e16b4876bd6538abd1f0c06
        - notebook
  

  • If downloading ERA5 data, need to change the pd.date_range end date to “2020-04-30”
  • I notice IceNet also has an ORA5Downloader() class. It would be great to include an example of how to use this in the notebook, since the notebook mentions that ORA5 data are used
  • The anchor links in the Highlights heading do not seem to work

• Does the notebook build and run in binder? mybinder repo

• Are all data sources openly accessible and properly cited (e.g. with citation to a persistent DOI) in the heading section?

  DOIs are missing for the specific OSI-SAF, ERA5 and ORA5 data used in the notebook

Pedagogy

• Are the notebook purpose and highlights clear?

  Yes, purpose is clear, as well as clear disclaimers as to how this version differs from that introduced by Andersson et al.

• Does the notebook demonstrate some specific data analysis or visualisation techniques?

  Yes, the notebook demonstrates advances in the development of an operational sea ice forecasting tool using deep learning. The workflow is complete with data downloaders and pre-processing tools, as well as training and inference steps for daily sea ice concentration prediction. The authors note that some of the more data-intensive components of the workflow have been withheld due to Binder's computational/memory limits. Improvements could be made in terms of prediction error assessment (see comment below)

• Is the notebook well documented, using both markdown cells and comments in code cells?
• Does the conclusion section provide clear and concise final say on the tools, analysis and/or datasets used?

  Yes, the Summary heading highlights the main steps/functionalities of the notebook, and the Extended usage section provides some useful resources for the interested user

• Is the notebook narrative well written (it does not require editing for structure, language, or writing quality)?

Ethical

• Is any linkage of datasets in the notebook unlikely to lead to an increased risk of the personal identification of individuals?
• Is the notebook truthful and clear about any limitations of the analysis (and potential biases in data and/or tools)?

  The original U-Net model introduced by Andersson was trained to do classification tasks (predict local monthly-mean sea ice extent), however the model here is refactored to a regression task (predict local daily sea ice concentration). It is unclear whether this new model would have the same predictive skill as the original model over the lead times explored in Andersson et al. Perhaps this is outside the scope of this notebook, but it would be interesting to see if you use the new model to predict an entire month of daily SIC (from which you can get daily SIE and then compute a monthly-mean), does it achieve similar skill to the original U-Net model's prediction to monthly SIE? I appreciate that for computational reasons the model in this notebook is limited in terms of training and inputs, but I think it would be useful to quantify what these limitations have on the prediction skill - especially if someone did want to use this for operational sea ice forecasting.

• Is the notebook unlikely to lead to negative social outcomes, such as (but not limited to) increasing discrimination or injustice?

Other Requirements

• All mentioned software should be formally and consistently cited wherever possible.
• Acronyms should be spelled out upon first mention.
• Licence conditions on images and figures must be respected (Creative Commons, etc.).

[annefou]

@bnubald the two reviewers suggested a few changes. Would you have time to have a look and implement their suggestions? Many thanks.

[bnubald]

Thank you @weiji14, @William-gregory, @annefou, I will go through them, I might be a bit delayed (2-3 weeks) due to focusing on a conference.

[annefou]

Hi @bnubald, just checking with you. How is it going? Any progress with the review? Many thanks.