Skip to content

Multi yoy no label#498

Merged
martin-springer merged 23 commits into
NatLabRockies:developmentfrom
cdeline:multi_YoY_no_label
Jun 24, 2026
Merged

Multi yoy no label#498
martin-springer merged 23 commits into
NatLabRockies:developmentfrom
cdeline:multi_YoY_no_label

Conversation

@cdeline

@cdeline cdeline commented Mar 24, 2026

Copy link
Copy Markdown
Collaborator
  • Code changes are covered by tests
  • Code changes have been evaluated for compatibility/integration with TrendAnalysis
  • New functions added to __init__.py
  • API.rst is up to date, along with other sphinx docs pages
  • Example notebooks are rerun and differences in results scrutinized
  • Updated changelog
  • Update degradation and multi-YoY notebook example to remove label=
  • Remove old degradation_timeseries_plot_old function cruft

@cdeline cdeline marked this pull request as draft March 24, 2026 21:44
@cdeline cdeline marked this pull request as ready for review March 24, 2026 23:16
@cdeline

cdeline commented Mar 24, 2026

Copy link
Copy Markdown
Collaborator Author

OK, so it looks like taking the median of every slope whose center-point falls within the moving window will re-create the original pd.rolling() approach for the standard (non-multi-yoy) case. Another change is that the new approach effectively labels the datapoints with label=center while the old approach is defaulting to label=right. This could be fixed if we wanted to go back to the original implementation.
image

@cdeline

cdeline commented Mar 24, 2026

Copy link
Copy Markdown
Collaborator Author

.. But with the multi-YoY output, taking the median of all slopes rather than a pointwise average will result in much more smoothing in the center of the timeseries (where you have more slopes available), but more variability at start and end of the timeseries when you're only averaging the 1-yr slopes.
image

@martin-springer

Copy link
Copy Markdown
Collaborator

.. But with the multi-YoY output, taking the median of all slopes rather than a pointwise average will result in much more smoothing in the center of the timeseries (where you have more slopes available), but more variability at start and end of the timeseries when you're only averaging the 1-yr slopes. image

Agreed, with the multi-yoy approach the time series plot becomes less meaningful as it kind of converges to the overall system degradation rate in the center of the plot and that's kind of unrelated to the years shown in the x-labels. How about, for plotting purposes, we only use the 1-year slopes (slopes <1.5 year to account for leap years) for the whole plot? Would this give us a better representation of the system degradation along the time axis?

@cdeline

cdeline commented Mar 27, 2026

Copy link
Copy Markdown
Collaborator Author

We could leverage the rolling_days input parameter - like filter out slopes that are longer than > (rolling_days *1.1). Then you should get similar plots whether you use multi-yoy or regular.

@cdeline

cdeline commented Apr 3, 2026

Copy link
Copy Markdown
Collaborator Author

OK, I've added a filtering step where only slopes <= 2 years + 1 day (731 days) are included in the median. This helps to avoid over-smoothing of the multi-YoY from having too many slope values. I think this is a good compromise. I decided against making the filtering based on the width of the moving window because you could quickly filter out all valid slopes.
image

@mdeceglie

Copy link
Copy Markdown
Collaborator

would this give us the behavior we're after? cdeline#1

@cdeline @martin-springer let me know what you think.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the year-on-year (YoY) degradation workflow to remove reliance on label=-based timestamping, adds explicit YoY timestamp metadata (YoY_times) to the degradation_year_on_year outputs, and refactors the degradation time-series plotting path to better support multi-YoY results.

Changes:

  • Removes label=-specific tests/fixtures and updates TrendAnalysis defaults to no longer pass label through yoy_kwargs.
  • Extends YoY calc output metadata with calc_info['YoY_times'] and updates degradation_timeseries_plot to use timestamps/durations from that structure.
  • Updates v3.2.0 changelog entries to reflect the new YoY_times output and the revised multi-YoY time-series plotting approach.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
rdtools/test/plotting_test.py Removes label-based coverage and updates multi-YoY handling assertions for degradation_timeseries_plot.
rdtools/test/degradation_test.py Removes tests for now-removed/changed label='center' behavior.
rdtools/plotting.py Refactors degradation_timeseries_plot to consume YoY_times and handle multi-YoY smoothing differently.
rdtools/degradation.py Removes label documentation/logic and adjusts how YoY timestamps/indices are produced.
rdtools/analysis_chains.py Removes default yoy_kwargs={"label": "right"} so chains don’t depend on the label parameter.
docs/sphinx/source/changelog/v3.2.0.rst Updates changelog notes for YoY_times and the new multi-YoY plotting behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread rdtools/degradation.py
Comment thread rdtools/degradation.py
Comment thread rdtools/plotting.py Outdated
Comment thread rdtools/plotting.py Outdated
Comment thread rdtools/test/plotting_test.py Outdated
@cdeline

cdeline commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator Author

Mike's PR was good. I do want to update the rolling median though to have the min_periods=rolling_days//4 instead of //2. I think this is a minor fix that we can include in this RdTools revision to make the time-series plot more resilient to small outages without losing fidelity. The API is still backwards compatible, the new plots will include more detail though if segments were previously excluded due to insufficient data length.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 9 changed files in this pull request and generated 8 comments.

Comment thread rdtools/degradation.py
Comment thread rdtools/degradation.py Outdated
Comment thread rdtools/degradation.py
Comment thread rdtools/plotting.py Outdated
Comment thread rdtools/plotting.py
Comment thread rdtools/analysis_chains.py
Comment thread rdtools/test/plotting_test.py Outdated
Comment thread rdtools/analysis_chains.py
@cdeline

cdeline commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator Author

OK, good to go @martin-springer and @mdeceglie.

@mdeceglie

Copy link
Copy Markdown
Collaborator

I do want to update the code to have the min_periods=rolling_days//4 instead of //2. I think this is a minor fix that we can include here to make the time-series plot more resilient to small outages without losing fidelity.

Since this would change the math relative to our latest release, I think the best path would be to make it an argument, we can change the default behavior in the next major version.

@martin-springer

Copy link
Copy Markdown
Collaborator

I do want to update the code to have the min_periods=rolling_days//4 instead of //2. I think this is a minor fix that we can include here to make the time-series plot more resilient to small outages without losing fidelity.

Since this would change the math relative to our latest release, I think the best path would be to make it an argument, we can change the default behavior in the next major version.

Commit 15918a3 adds the new argument min_periods_divisor. If it's not set it defaults to 2 but raises a FutureWarning that the default will change to 4 in the next release. Documented with Issue #514

@martin-springer martin-springer merged commit f4a18e1 into NatLabRockies:development Jun 24, 2026
19 checks passed
martin-springer added a commit that referenced this pull request Jul 1, 2026
* Multi yoy integration - combines PRs #456, #460 and #464 at once. (#470)

* add keyword 'label' to degradation_timeseries_plot, enabling 'left' and 'center' labeling options.

* Update changelog, add pytests, update sphinx documentation

* fix flake8 grumbles

* update pytests to include axes limits

* fix flake8 grumbles

* add 'label' input option to `degradation_year_on_year`. Fixes #459

* add pytests and update changelog.

* flake8 grumbles

* Minor updates to setup.py (constrain scipy<1.16) and refactor degradation_test

* Custom fix for Pandas < 2.0.0 which can't average two columns of timestamps.

* flake8 grumbles

* keep TZ-aware timestamps.  Update pytests to specifically test _avg_timestamp_old_Pandas

* flake8 grumbles

* try to UTC localize the pytest...

* Add .asfreq() to get pytests to agree

* switch to calendar.timegm to hopefully remove TZ issues..

* regardless of uncertainty_method, return calc_info{'YoY_values')

* update _right dt labels to correct _left labels in degradation_year_on_year

* update _avg_timestamp_old_Pandas to allow for numeric index instead of timestamp

* add left label option to degradation_year_on_year

* update degradation_year_on_year, index set to either left, center or right. Consistent with #394 - multi_yoy

* update return for default = none uncertainty option

* degradation_year_on_year - go back to single return when uncertainty_value = None to avoid breaking pytests.

* add multi-year aggregation of slopes in degradation_year_on_year

* add multi_yoy kwarg in degradation_year_on_year to toggle the multi-YoY function.

* update plotting for detailed=True, allow usage_of_points > 2

* flake8 grumbles

* update plotting detailed=True for  (even) and (odd) number of points coloring

* To allow multi_yoy=True in plotting.degradation_timeseries_plot, resample.mean() the YoY_values.

* flake8 grumbles

* Add warning to degradation_timeseries_plot when multi_YoY=True

* update to warning message in plotting.degradation_timeseries_plot

* fix flake8 grumbles

* nbval fixes from qnguyen345-bare_except_error

* Add pandas 3.0 futurewarning handling

* Try again to solve pandas3.0 futurewarning

* attempt 3 to fix nbval

* Add infer_objects to remove futurewarning

* minor inline comment update

* update plotting tests to be relative value, update ordering of module import in plotting.py, per Copilot review.

* update inline comments and whatsnew docs

* Clean up inline comments per Copilot review

* added multi-YoY pytest - still need to catch warnings

* Add a warnings.catch_warnings to the plotting pytest

* flake8 grumbles

* add multi-YoY=True pytest

* updated changelog

* update changelog

* implement copilot suggestions

* linting

* use s instead of ns for pandas 3 compatibility

* update pandas version comparison

* set matplotlib non-gui backend for tests

* set dtype resolution based on pandas version

* linting import order

* boost degradation.py test coverage

* update changelog

* exclude coverage reports with suffixes

* Update rdtools/degradation.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* remove label=None handling, rely on default 'right' behavior

* refactor dt_center  tz handling for old pandas

* simplify _avg_timestamp_old_Pandas

* degradation_timeseries_plot: change rolling median min_periods to rolling_days / 4.

* remove degradation_timeseries_plot(label=) and just default to center=True

* update sensor_analysis() and clearsky_analysis() docstrings to discuss passing `label=right` kwargs

* flake8 updates

* Initial commit - multi-YoY notebook

* pretty-print notebook dataframes with tabulate.

* update notebook requirements to silence pandas warnings

* add multi-yoy nb to tutorials

* fix nblink path

* Change the yoy_values index to be named 'dt'.  Add new illustrations at the end of the multi-YoY notebook.

---------

Co-authored-by: Michael Deceglie <mdeceglie@users.noreply.github.com>
Co-authored-by: martin-springer <martinspringer.ms@gmail.com>
Co-authored-by: Martin Springer <97482055+martin-springer@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Increase test coverage (#486)

* add keyword 'label' to degradation_timeseries_plot, enabling 'left' and 'center' labeling options.

* Update changelog, add pytests, update sphinx documentation

* fix flake8 grumbles

* update pytests to include axes limits

* fix flake8 grumbles

* add 'label' input option to `degradation_year_on_year`. Fixes #459

* add pytests and update changelog.

* flake8 grumbles

* Minor updates to setup.py (constrain scipy<1.16) and refactor degradation_test

* Custom fix for Pandas < 2.0.0 which can't average two columns of timestamps.

* flake8 grumbles

* keep TZ-aware timestamps.  Update pytests to specifically test _avg_timestamp_old_Pandas

* flake8 grumbles

* try to UTC localize the pytest...

* Add .asfreq() to get pytests to agree

* switch to calendar.timegm to hopefully remove TZ issues..

* regardless of uncertainty_method, return calc_info{'YoY_values')

* update _right dt labels to correct _left labels in degradation_year_on_year

* update _avg_timestamp_old_Pandas to allow for numeric index instead of timestamp

* add left label option to degradation_year_on_year

* update degradation_year_on_year, index set to either left, center or right. Consistent with #394 - multi_yoy

* update return for default = none uncertainty option

* degradation_year_on_year - go back to single return when uncertainty_value = None to avoid breaking pytests.

* add multi-year aggregation of slopes in degradation_year_on_year

* add multi_yoy kwarg in degradation_year_on_year to toggle the multi-YoY function.

* update plotting for detailed=True, allow usage_of_points > 2

* flake8 grumbles

* update plotting detailed=True for  (even) and (odd) number of points coloring

* To allow multi_yoy=True in plotting.degradation_timeseries_plot, resample.mean() the YoY_values.

* flake8 grumbles

* Add warning to degradation_timeseries_plot when multi_YoY=True

* update to warning message in plotting.degradation_timeseries_plot

* fix flake8 grumbles

* nbval fixes from qnguyen345-bare_except_error

* Add pandas 3.0 futurewarning handling

* Try again to solve pandas3.0 futurewarning

* attempt 3 to fix nbval

* Add infer_objects to remove futurewarning

* minor inline comment update

* update plotting tests to be relative value, update ordering of module import in plotting.py, per Copilot review.

* update inline comments and whatsnew docs

* Clean up inline comments per Copilot review

* added multi-YoY pytest - still need to catch warnings

* Add a warnings.catch_warnings to the plotting pytest

* flake8 grumbles

* add multi-YoY=True pytest

* updated changelog

* update changelog

* implement copilot suggestions

* linting

* use s instead of ns for pandas 3 compatibility

* update pandas version comparison

* set matplotlib non-gui backend for tests

* set dtype resolution based on pandas version

* linting import order

* boost degradation.py test coverage

* add tests for error handling in analysis_chain

* update changelog

* update changelog

* exclude coverage reports with suffixes

* Update rdtools/degradation.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* remove label=None handling, rely on default 'right' behavior

* refactor dt_center  tz handling for old pandas

* simplify _avg_timestamp_old_Pandas

* degradation_timeseries_plot: change rolling median min_periods to rolling_days / 4.

* remove degradation_timeseries_plot(label=) and just default to center=True

* update sensor_analysis() and clearsky_analysis() docstrings to discuss passing `label=right` kwargs

* flake8 updates

* Initial commit - multi-YoY notebook

* pretty-print notebook dataframes with tabulate.

* update notebook requirements to silence pandas warnings

* add multi-yoy nb to tutorials

* fix nblink path

* Change the yoy_values index to be named 'dt'.  Add new illustrations at the end of the multi-YoY notebook.

* clean up pending changelog

---------

Co-authored-by: cdeline <chris.deline@nrel.gov>
Co-authored-by: Michael Deceglie <mdeceglie@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* rename pending to v3.2.0

* Update docstrings and address simple comments from copilot

* flake8 grumbles

* Fix broad `except ValueError` in `degradation_timeseries_plot` for multi-YoY detection (#490)

* Initial plan

* Replace broad except ValueError with explicit duplicate-index check in degradation_timeseries_plot

Co-authored-by: cdeline <23244088+cdeline@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: cdeline <23244088+cdeline@users.noreply.github.com>

* Multi yoy no label (#498)

* updated degradation_timeseries_plot as version2 for now

* fix flake8 grumbles

* avoid pandas error "ValueError: Other Series must have a name"

* Redo degradation_timeseries_plot - include in median if dt_center is within rolling window.

* remove label= functionality. Update Multi-year_on_year_example.ipynb

* Re-run Multi-year_on_year_example.ipynb, check for text changes.

* update analysis_chains to remove `label=`. Update whatsnew

* simplify multi-YoY timeseries plotting by filtering to slopes <= 2 years, removing warning.

* update whatsnew

* code cleanup

* Revision of degradation_timeseries_plot

* Bug fixes

* Update notebooks to include `center=True`

* Update nbval to include new notebook

* add new kwarg to TrendAnalysis.plot_degradation_timeseries

* Update plotting.py timeseries plot docstring.  Also set pd.rolling(min_periods=rolling_days//4)

* fix copilot documentation review recommendations; add plotting pytests

* add min_periods_divisor argument

* update pvdaq_system_4 url (#513)

* update pvdaq_system_4 url

* changelog

* revert unnecessary changes in notebooks

* update changelog

* re-run notebooks

---------

Co-authored-by: Michael Deceglie <Michael.Deceglie@nrel.gov>
Co-authored-by: martin-springer <martin.springer@nrel.gov>
Co-authored-by: Martin Springer <97482055+martin-springer@users.noreply.github.com>

* re-run TrendAnalysis nb

* comments mdeceglie

* update release date

---------

Co-authored-by: Chris Deline <chris.deline@nrel.gov>
Co-authored-by: Michael Deceglie <mdeceglie@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: cdeline <23244088+cdeline@users.noreply.github.com>
Co-authored-by: Michael Deceglie <Michael.Deceglie@nrel.gov>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants