Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
325 lines (233 sloc) 20.2 KB

Report for GSOC 2019

This page is the location for the reporting related to the GSoC 2019 project titled A next-generation GUI for visualizing big gridded data in Python with the organisation Earth Science Information Parteners and member organisation US Geological Survey.

Student: Harman Deep Singh Mentors: Rich Signell, Martin Durant ESIP POC: Annie Bryant Burgess

Project Details
Initial Proposal Available here
Repository intake/xrviz
Documentation xrviz.readthedocs.io
Milestones #1, #2, #3, #4
Duration ~4 Months (6 May'19 - 26 Aug'19)

Index

Community Bonding Period (May 6th - May 26th)

Community Bonding Period Milestone

Activity:

  1. Introduced myself and this project to the ESIP, Intake and Pyviz community.

  2. Established communication channel and had web meeting with mentors for discussion regarding workflow of the project.

  3. Took overview of other catalog systems available. Watched ESIP Tech Dive by Chris Holmes on STAC.

  4. Watched ESIP Tech Dive talk on Intake.

  5. Wrote the post Beginning of my GSOC Journey! (Issue #4).

  6. Set up development environment.

  7. Tested use of catalogs with earlier dashboards and discussed Issue #3724 · pyviz/holoviews.

  8. Read and executed the following documentations (Issue #3):

  9. Explored Pangeo Datasets and Catalogs (Issue #7).

  10. I have also added revising OOP Python concepts to my studylist.

  11. Prepared quick basic implementation of data description section (Issue #5).

Next week's Agenda:

  • Discussion regarding design of XrViz.
  • Preparation of data input section.
  • Learn using Pytest, which is going to be the testing framework for XrViz.

Coding Period (May 27 - Aug 26)

Week 1 (May 27 - June 2)

Weekly Activity

  1. Took an overview of Pytest.
  2. Understood working of dfviz/widgets.py.
  3. Weekly e-meet: The following topics were discussed:
  1. Started implementation of Display and description section (PR#2). Current status:

    1.Dimensions Display: Dimensions

    2.Coordinates Display: Coordinates

    3.Variables Display: Variables

    4.Attributes Display: Attributes

Week 2 (June 03 - June 09)

Weekly Activity

  1. Weekly e-meet: The following topics were discussed:
    • Appearance and functionality of display section. After discussion we concluded to have a simpler interface, displying only the variables in the selector. Other global properties would be always visible.
    • Overview and usage of tests.
    • Features and functionality of Fields panel.
  2. Implemented simpler interface consisting of Display and description section (PR #2). For more details please refer Issue #14 and Issue #15. Current status:

Simpler Interface 3. Started implementation of Basic Fields and Output Panel (PR#3). Current status:

Basic Fields and Output Panel

Week 3 (June 10 - June 16)

Weekly Activity

  1. Improved the Basic fields and output panel. Added support for plotting data coordinates with respect to variables, which earlier resulted in errors.
  2. Completed implementation of Basic Fields and Output Panel (PR #3). The plotting function has been reimplemented using sel method.
  3. The following topics were discussed in weekly e-meet:
    • ignore_events method present in dfviz, can be useful for some interactions in xrviz.
    • Demonstrated the working of interface.
    • convert_widget function.
    • Way to set/reset variables to coordinates.
    • Aggregation methods (Issue #21)
    • Overview of tasks for Phase 2:
  4. Started implementation of Aggregations in the fields panel (PR #4). Current Status:
  • For single selection

Aggregations 1 selection

  • For multiple selections

Aggregations 1 selection Aggregations 1 selection

  1. Started discussion to find out a possible solution to set and reset coordinates within the interface. For more details please refer Issue #20.

Week 4 (June 17 - June 23)

Weekly Activity

  1. Improved the functionality implemented earlier for Aggregations in the Fields panel. Now the user can aggregate along remaining dimensions (not selected in x and y ). The option to create a Select widget or Animate is also present in case the user does not wish to apply any aggregation on the dimension. (PR #4)

Aggregations

  1. Implemented functionality to set reset coords (PR #7). User can select and set any variable as a coordinate.

Set Reset Coords

  1. Modified set reset coords to work with aggregations. The user can have aggregation for a variable along a dimension, even when x and y Select has coord as options.

Set Reset Coords

  1. Weekly e-meet: The following topics were discussed:
  • Installation package (PR #8588 · conda-forge)
  • Demonstration of Aggregations (PR #4)
  • Demonstration of Set reset Coords (PR #7)
  • Discussed Tabular Rearrangement (Issue #26)
  • Generalisability of the interface: In future the dashboard can be used as base by the users, or community to visualise data related to a particular domain such as Astronomy, Oceanography by adding domian specific customisations.
  1. XrViz can be installed as a package by conda install -c conda-forge xrviz. (conda-forge/xrviz-feedstock)

Week 5 (June 24 - June 30)

Weekly Activity

  1. Rearranged panels into Tabs (PR #12). Tabs

  2. Started implementation of Style Panel (PR #13). Style

  3. Started implementation of Projection Panel (PR #14). Projection

  4. Weekly e-meet: The following topics were discussed:

    • Demonstrated interface after Tabular rearrangement.
    • Demonstrated Style Panel.
    • Demonstrated Projection Panel.
    • Changed topic for blogpost to “XrViz: An interactive visualisation interface for Xarrays” (Issue #9).
    • Enable Travis CI (PR #15).
  5. Modified Style Panel according to the changes discussed.

  6. BlogPost XrViz: An interactive visualisation interface for Xarrays.

  7. Implemented xrviz.example(), so the following command can be used to try the interface(PR #20).

python -c "import xrviz; xrviz.example()"
  1. Fixed pip installation( PR #18), which solved Colormap not found if installed from pip (Issue #17).

Week 6 (July 01 - July 07)

Weekly Activity

  1. Completed implementation of Style Panel (PR #13).
  2. Modifed Projection Panel according to changes discussed in last e-meet (PR #14).

With show_map (basemap): Projection

Without show_map (projection): Projection

  1. Weekly e-meet: The following topics were discussed:
    • Demonstrated changes made in Style Panel.
    • Demonstrated changes made in Projection Panel.
    • Tests to be implemented for Style, Projection and Dashboard.
    • Use of Coverage.

Week 7 (July 08 - July 14)

Weekly Activity

  1. Started implementation of Series Extraction (Issue #24), (in branch pre_series, yet to create PR).
  2. Weekly e-meet: The following topics were discussed:
    • Demonstrated changes made in Projection Panel
    • Passing of parameters to select values upon initialisation
    • Replace central_latitude and central_longitude widgets with LiteralInput.
    • Demonstrated recently started Series extraction feature.
  3. Completed implementation of Projection Panel (PR #14).

Projection

  1. Started adding more tests (in branch pre_tests_for_style_proj, yet to create PR).

Week 8 (July 15 - July 21)

Weekly Activity

  1. Weekly e-meet: The following topics were discussed:
    • Minor changes to be made
      • Rasterize in style panel
      • Textbox for crs and projections params instead of LiteralInput
      • Improve styling of displayer in Fields Panel
    • Demonstrated progress of Series Extraction feature.
    • Change of blogpost topic: "Amazing Pyviz" -> "Sigslot"
    • Good time to start documentation, will use readthedocs.
  2. Minor changes added with Change displayer width policy to min (PR #25) and Urgent fixes (PR #28).
  3. Added Tests (PR #26), with coverage of 97%.
  4. Improved Series extraction (PR #27) by adding feature to extract along 2d coordinates and display information of other dimensions in hover.

Series

  1. Created Explore-HRRR-with-XrViz, to demonstrate the progress of XrViz. Also deployed this notebook on binder.

Week 9 (July 22 - July 28)

Weekly Activity

  1. Completed Series extraction (PR #27).
  2. Weekly e-meet: The following topics were discussed:
    • Structure of and topics to cover in documentation.
    • Possible reasons behind faulty behaviour with COAWST dataset.
    • Presentation of XrViz to ESIP and Anaconda.
  3. Published a blog post Sigslot — A simple signalling mechanism for Panel.
  4. Published a blog post Let’s visualise big gridded data with Intake and XrViz.
  5. Started writing documentation using Sphinx and Read the Docs (PR #29).

Week 10 (July 29 - Aug 04)

Weekly Activity

  1. Weekly e-meet: The following topics were discussed:
    • The documentation should be short, clear and concise.
    • Schedule for presentation of Xrviz to Anaconda and ESIP.
  2. Completed Initial Documentation (PR #29). Docs are now available at xrviz.readthedocs.io.
  3. Started improving documentation further (PR #32).
  4. Presented Xrviz in Anaconda Demo Friday.

Week 11 (Aug 05 - Aug 11)

Weekly Activity

  1. Completed improving documentation (PR #32).
  2. Improved README by adding badges and basic information (PR #35).
  3. Started looking into solution for issue: quantile doesn't work for dask-powered xarrays (Issue #31) and attemped solving it in PR #37.

Week 12 (Aug 12 - Aug 18)

Weekly Activity

  1. Weekly e-meet: The following topics were discussed:
    • The dramatic slowness of XrViz when run within Intake (PR #398 intake)
    • GSOC Report and code submission.
    • Timings for presentation of XrViz to Pangeo.
  2. Presented Xrviz to Pangeo in Weekly Meeting. A 10 minute demonstration of the interface.
  3. Completed PR #37. Now users can also visualise dask xarrays.

Current State of the Project

The work done during the coding period has laid down a foundation for further development of XrViz. It is well positioned to grow and thrive, embedded in the intake project. The documentation would provide a good starting point for future contributors to get acquainted with the interface. The feedback I've received so far from the community indicates that they find this as a valuable contribution.

Merged Pull Requests: #2, #3, #4, #7, #12, #13, #14, #15, #18, #20, #24, #25, #26, #27, #28, #29, #32, #35, #37

Future work

I feel that we have achieved most of what we set out to do in terms of deliverables. However, the following additions will make the interface more useful:

  1. Integration of Xrviz in Intake GUI.
  2. A way to launch the dashboard directly from command line as done in ncview.
  3. The dashboard can be customised to visualise data related to a particular domain such as Astronomy, Oceanography by adding domian specific operations and functionality.

Lessons learned

Working with ESIP, Intake and HoloViz under Google Summer of Code has been an amazing and rewarding experience. It has provided me a chance for learning and self improvement. Here are some of the lessons I have learnt:

  1. Creating interactive GUI with Panel and dashboarding. Well designed GUIs have the potential to simplify the workflow and save user time.
  2. The importance of automated testing in software development. Pytest and Travis CI has been used in this project.
  3. Creating and writing documentation. Clear and concise documentation requires careful thinking.
  4. Routine code submission, discussion and review by mentors helps completing task in short time and in directed manner.
  5. Became more familiar with Git, Github and open source culture and software development in general. Also learnt several python best practices. This would be helpful in contributing to other projects.
  6. This project increased my interest in events and developments of earth science community.
  7. The importance of effective communication in open source. This is the key to explain your ideas and get help from others on the issues being faced. Weekly e-meets helped me gain more confidence and ability to express myself better.
  8. Always staying open to new ideas and suggestions is vital. It helps you to understand the viewpoint of mentors, community and any other person in general.

Acknowledgment

I am extremely grateful to my mentors Rich Signell and Martin Durant for helping me out at various stages throughout the summer. Special thanks to Annie Burgess, ESIP Lab Director for facilitating the Summer Program. Sincere gratitude to Google for offering me this great opportunity.

You can’t perform that action at this time.