Releases: ODM2/ODM2DataSharingPortal
Release 0.17.1
This patch release addresses several issues reported and patched as hotfixes following the release of 0.17.0.
Bugfixes
- Fix 500 errors on a handful of pages (#704, #707, #710, #712)
- Fix new user site registration issue (#708, #709)
- Restore CSV download feature (#713)
Full Changelog: v0.17.0...v0.17.1
v0.17.0 Organization Site Ownership
New in v0.17.0 Organization Site Ownership
This release is a data restructure and subsequent updates to application code to allow ownership of sites/samplingfeatures by organization rather than individual accounts.
New Features
- Restructures underlying data from user based ownership to organizational based owner (see more details below).
- Adds database support (by UI or frontend tooling) for users to be affiliated multiple organizations.
- Adds ability for users to transfer sites between affiliated organizations and individual accounts.
- Restored admin's ability to transfer sites to any account (a previous existing feature that has been broken).
- Quality of life improvements for upcoming changes pop-up.
- Make resize more dynamic and based off page size.
- Add the ability for users to close popup without scrolling to the bottom.
- Add cookie based check to not display the pop-up for 24 hours after it has been closed.
- Improves browse sites load performance with updates query to load data.
Summary of how organizational ownership works
This release changes the underlying data structure from having individual accounts own sites to ownership through organizations. This change is necessary to support and unlock new features such as multi-account governance of sites and better management of sites for organizations.
What happens to sites associated with my account?
All users will have an individual organization generated and affiliated with their account. Any sites that were solely associated with your account (i.e. no prior organization associated) will automatically be transfers to this organization. Any sites associated with an organization, will be transferred to that organization.
Why can I suddenly view a bunch of extra sites?
If your account is affiliated with multiple organizations, those organizations sites will now be displayed on the my sites page. Sites will be group by affiliation, so that you can still tell which sites are associated with your individual organization and which are through organizational affiliations.
image
How can I add or remove accounts that are affiliated with my organization?
As of this release there is no interface for organization management, though we identify that as a priority feature and will track that development through a separate milestone. If you need organization management you can contact support at help@monitormywatershed.org
Full Changelog: v0.16.0...v0.17.0
v0.16.0 StreamWatch Schools Enhancements
New in v0.16.0 StreamWatch Schools Enhancements
This release contains a series of improvements to further support the StreamWatch Schools program.
New Features
- Updates StreamWatch schools logo in several locations throughout the application.
- Revises the StreamWatch Schools section of the home page, clarifying instructions and improving navigation.
- Adds new query parameters to the browse sites page to allow users to preselect specific dataTypes (programs), organizations, and siteTypes (site types). If you wish to select multiple dataTypes, organizations, or siteTypes, pass them as comma delimited values in the corresponding query parameter.
- Adds filterable search feature to assessor select element on main StreamWatch Schools form
- Adds support for site photo uploads to StreamWatch Schools form
- Adds non-detect and range validation to chemical constituents
- Adds new macro invertebrates subform to the StreamWatch Schools form
- Updates StreamWatch summary table to include select parameters
Acknowledgements
This work was supported by contracts from The Watershed Institute.
Want more details?
Check out the issues and pull requests tagged to the milestone for this release.
StreamWatch Enhancements
Full Changelog: v0.15.0...v0.16.0
v0.15.0 Authentication Improvements
New in v0.15.0 Authentication Improvements
This release marks an important milestone towards improving the application's user authentication and authorization. The release completely replaces the original user account management approach in favor of a more secure and flexible AWS Cognito based approach and lays the groundwork for a variety of user account related improvements.
Change Log Highlights
New Features
- Adds a
SteamWatchfilter option under the Programs heading on the Browse Sites page. This allows users to more easily identify and view sites with StreamWatch data. - Adds anchor locations for programs on home page for quicker access. You can now navigate to the program locations by adding program names to the url. Program names supported are
#EnviroDIY,#LeafPack, &#StreamWatch
(e.g. https://monitormywatershed.org#StreamWatch) - Adds CSV export support for StreamWatch data
- Adds StreamWatch Schools to banner on home page, and several other locations
Improvements
- Complete refactor of authentication to utilize AWS Cognito. This provides better security for user account information and addresses some long standing issues with password reset. This also paves the way for future enhancements to user accounts and permissions.
- Refactors the CSV Download endpoint to improve performance. Testing showed an approximately 25% increase in performance.
Bug Fixes
- Fixes bug that prevented user's from being able to see StreamWatch data for sites they don't own. [Already live as hotfix]
Acknowledgements
This work was supported by contracts and grants from the Stroud Water Research Center, The Watershed Institute and the AWS IMAGINE Grant Program
Want more details?
Check out the issues and pull requests tagged to the two milestones under this release.
StreamWatch Customization
v0.15 release - Refactor Authentication using Cognito
Full Changelog: v0.14.0...v0.15.0
v0.14 release: StreamWatch Schools Program
New in v0.14 release: StreamWatch Schools Program
This important release introduces support for The Watershed Institute’s StreamWatch volunteer monitoring network by developing data entry forms for the StreamWatch Schools program. This is an initial step in providing a more flexible set of options for sharing a wider range of stream and lake monitoring data, from water sample laboratory data to habitat assessments, and for additional monitoring networks.
This work was supported with funding from the The Watershed Institute.
New Features
- Data entry forms for StreamWatch Schools program.
- Inclusion of the StreamWatch network on the Monitor My Watershed landing page, joining EnviroDIY and the Leaf Pack Network.
- Inclusion of The Watershed Institute as a Monitor My Watershed "Partner" on the footer of every page, as a pilot for expanding the reach and utility to many other networks and partners.
Improvements
- Backend development of generalized “observations” that fully leverages the power of the Observations Data Model v2.0 (ODM2), to enable a flexible system for users to record data for a wide array of measurement types and variables.
- Backend development of categorical “observations” using ODM2, to support any categorical value that might be used in habitat assessments, etc.
- Continued work on "paying off" technical debt and removing cruft.
- Modernized and refactored all code that we touch (but much old code still remains!)
- Rewrote time zone handling to use the new
zoneinfoPython 3 standard library that will be exclusively supported by Django 4, rather thanpytz(#617)
- Updated Tech Stack on database server to supported software versions
Bug Fixes
- Hotfixes to deploy scripts that automatically close all open and idle database connections every night, caused by remaining poorly-written legacy code, to prevent compute resource drain over time (that had previously led to database server crashes).
For the full list of issues that we addressed with this release, see this Milestone on GitHub:
v0.13.0: Tech Debt Upgrades and TSV Improvements
New in v0.13.0: Tech Debt Upgrades and TSV Improvements
This important release reflects an important step in our year-long effort to address technical debt by updating the software stack dependencies and refactoring the codebase to remove cruft, to more efficiently use the capabilities of the updated stack, and to facilitate future feature development and maintenance. This release also improves the Time Series Visualization (TSV) functionality, which was first launched with v0.12.0 as a replacement for the outdated Time Series Analyst (TSA) codebase.
This work was supported by contracts from the Stroud Water Research Center and The Watershed Institute.
Improvements
- Continued addressing technical debt and cruft removal.
- Moved remaining 'dataloader' tables from a second separate database into a single database.
- Replaced some raw SQL with new ODM2 ORM data models
- Removed dead code related to Influxdb
- Cache busting to eliminate the need for users to clear their browser caches after a new release (#529)
- Water One Flow (WOFpy) web service landing page updated
- Station data now upload correctly formatted data when one variable UUID is incorrect or empty (#303, #556)
- Updated database server to the new AWS EC2
C6iinstance type, for better performance, including doubling the bandwidth to the web server (#587)
- Updated Tech Stack to supported software versions
- TSV Enhancements
- Better CSS to autofill and scale with page size
- More flexible zooming and added panning on plot
- Data point hover popup now always displays the year
- Performance enhancements through dynamic data loading
- Additional enhancements are described in issue #551 and other TSV-tagged issues.
Bug Fixes
- Address critical bugs
For the full list of issues that we addressed with this release, see this Milestone on GitHub:
v0.12.1: Hot Fixes to Database Connections and Error Handling
New in v0.12.1: Hot Fixes to Database Connections and Error Handling
This hot fix release improves the performance and stability of our new deployment on AWS by better managing database connections, more efficiently inserting data from device POSTs, and more accurately providing HTTP error codes.
This work was supported by a contract from the Stroud Water Research Center's general fund.
Improvements
- Faster API traffic response times, by at least 3x or more, for devices POSTing data at logging intervals. #547
- This improved performance was gained by replacing data insert code based on older Django models with newer code based on SQLAlchemy. These are first steps toward our next Milestone: Release 0.13 - Refactor code for performance & scalability
- Better error handling for device HTTP POST requests, providing more accurate HTTP error codes in server responses. #539, #541
Bug Fixes
- Stabilized system response times, due to CPU utilization maxing out over time from accumulating open database connections that were consuming server resources even though idle. #547
- Fixed aliasing for plots in the new Time Series Visualization (TSV). #536
- Replaced additional code that was related to the now deprecated Time Series Analyst (TSA) and was causing incorrect error code responses with some devices. #538, #539
For the full list of issues that we addressed, see these Milestones on GitHub:
v0.12.0: Update to Python 3.8 & Django 2.2; Migrate to AWS
New in v0.12.0: Update to Python 3.8 & Django 2.2; Migrate to AWS
This major release updates the entire software stack to supported versions and migrates to hosting by Amazon Web Services (AWS). It represents a substantial pay off of accumulated "tech debt" since 2018 and paves the way for feature development scheduled for 2022. A major goal of this effort was to improve the reliability and performance for users.
This work was supported by a contract from the Stroud Water Research Center's general fund.
Improvements
- Updating the codebase to more recent Long Term Release (LTR) versions of dependencies:
- Ubuntu Linux 16.04 to 20.04. #391
- Python 2.7 to 3.8. #381
- Django 1.11 to 2.2. #385
- PostgreSQL 9.5 to 12. #497
- For a complete list of updates, compare our old
requirements.txtto our current Condaenvironment.ymlfiles.
- Replaced all use of the InfluxDB time series database with the TimescaleDB extension to PostgreSQL. #434
- This substantially improves the reliability of the web application, while also simplifying the codebase, substantially reducing data redundancy, increasing performance, and massively reducing data storage to 5% of previous.
- Replaced use of Time Series Analyst (TSA) software with a new built-in Time Series Visualization (TSV) tool. #512, #515
- Although this presently reduces the features available for interactive visualization of time series data, it opens the door to new features that we will develop over the next year. TSV will also be more performant and reliable that TSA.
- Deployed https://monitormywatershed.org to AWS EC2 instances, for much improved reliability and performance.
Bug Fixes
- Fixed numerous bugs related to data visualization. #473, #489, #444
- Fixed numerous bugs related to CSV downloads and uploads. #364, #443, #430, and many others
- Fixed bug with Water One Flow web services (WOFpy), in which only one value was returned per request. #432
For the full list of issues that we addressed, see these Milestones on GitHub:
v0.11.1: Hotfixes
This minor set of hotfixes was released to production servers in March 2020.
It fixed a few bugs and bumped Waitress and Django dependancies to more secure versions.
For more details, see the Release 0.11.1 - Hotfixes milestone.
v0.11: Leaf Pack update + minor features & fixes
The ODM2DataSharingPortal repository contains code for a Python-Django web application enabling users to upload, share, and display data from their environmental monitoring sites. Data can either be automatically streamed from Internet of Things (IoT) devices, manually uploaded via CSV files, or manually entered into forms.
The ODM2 Data Sharing Portal is built on the Observations Data Model Version 2 (ODM2) information model and supporting software ecosystem, as described in the repo’s documentation.
The main instance of this application is for the Monitor My Watershed data sharing portal supporting the EnviroDIY and Leaf Pack Network communities.
New in v0.11: Leaf Pack update + minor features & fixes
This release focused on updating capabilities to support the new Leaf Pack Network aquatic macroinvertebrate data entry workflows and to calculate the new pollution sensitivity index values.
New Features
- Overhauled data entry forms and data visualization outputs for Leaf Pack Network experiments, to support the new Leaf Pack protocols. #404, #405, #406, #407, #408, #409
Improvements
- Moved "add new sensor" button above request notice. #412
- Added "Notes" field to manage sensors table. #415
- Updated controlled vocabularies for dropdown lists. #241
- Improved output of "View Token UUID List" to better match sorting and case used in ModularSensors example sketches. #437
- Security updates:
Bug Fixes
- Fixed bugs related to sparkline plots & Time Series Analyst (TSA) not connecting to the data. #436
For the full list of issues that we addressed, see our Release 0.11 - Leaf Pack update + minor features & fixes milestone on GitHub.