Detect sunny days

[wfvining]

The PVFleets QA project does this by fitting curves (quadratic for fixed systems, quartic for tracking systems) on each day.

[cwhanse]

We probably have to resolve what "sunny" means while these functions are implemented.

Is the scope of this issue to detect full days with mostly sunny conditions? Or more generally to detect periods of "sunny" conditions?

[wfvining]

The curve-fitting functions detect full days with mostly sunny conditions (the level of sunny-ness can be controlled by a parameter: the minimum r-squared for the fit). For a day where the curve fit is sufficiently good, every data point on that day will be marked True.

[cwhanse]

Sounds like "sunny" means a smooth parabolic shape to the GHI vs. time curve. If that's correct, "sunny" is related to but not equivalent to "clear sky" as used in pvlib.clearsky.detect_clearsky

[wfvining]

Yes, they seem related, although I don't totally understand the pvlib function. The funcitons from PVFleets don't just work on parabolic shapes, but also quartic shapes from tracking systems, and don't require any additional input (i.e. clearsky data, although they do require a day/night mask).

[cwhanse]

Related but different. The PVFleets function is anticipating a shape without expecting a magnitude, so it can be applied to power data. The pvlib function is expecting a smooth curve that is statistically similar to a clearsky irradiance curve, so applies only to global horizontal irradiance data.

[wfvining]

The PVFleets function is anticipating a shape without expecting a magnitude

Exactly. The magnitude can also be different from day to day, so if part of the system is taken offline for a few days, or an inverter is offline we can still identify sunny days. (not sure if I'm imagining realistic scenarios here)

[cwhanse]

Seems like a stretch to call the successes "sunny" days. Any day with a relatively smooth profile may pass the criteria, e.g., completely overcast, panels covered by snow.

[wfvining]

Good point. Adding to that for a tracking system, sunny days with a broken tracker will not be marked sunny.

[wfvining]

To address that, in the PVFleets functions there are certain minimums that must be met in addition to the curve fit. For fixed systems all data less than 0.4*data.mean() is discarded, and days with less than 5 hours of data remaining are marked as not sunny. For tracking systems all data less than 0.1*data.mean() is discarded, the same 5 hour criteria is applied, and any day where the maximum value is less than the data.mean() is marked as not sunny.

[wfvining]

I am looking at two paths forward with this implementation: a single function with a kwarg specifying fixed/tracking, or two separate functions. Here is a rough overview of what each would look like. There are some differences concerning which data is included in the curve fitting for the tracking vs fixed functions in the original PVFleets implementation that are pushing me towards the two-function option. The main difference is that the check for a tracking system requires a quartic and quadratic fit, but long periods of time (~9 hours) from morning/evening on each day are excluded from the quadratic fit to address problems with long tails caused by a stuck tracker (only night time/early morning/late evening are excluded in the check for a fixed system).

Single function API

def sunny_days(power_or_irradiance, daytime, correlation_min=0.94,
               fixed_max=0.96, tracking=False):
    """Return True for values on days that are sunny.

    Sunny days are identified when the :math:`r^2` for a quadratic fit
    to the power data is greater than `correlation_min`. If `tracking`
    is True then a quartic is fit instead of a quadratic.

    Parameters
    ----------
    power_or_irradiance : Series
        Timezone localized series of power or irradiance measurements.
    daytime : Series
        Boolean series with True for times that are during the
        day. For best results this mask should exclude early morning
        and evening as well as night. Morning and evening may have
        problems with shadows that interfere with curve fitting, but
        do not necessarily indicate that the day was not sunny.
    correlation_min : float, default 0.94
        Minimum :math:`r^2` for a day to be considered sunny.
    fixed_max : float, default 0.96
        Maximum :math:`r^2` for a quadratic fit when `tracking=True`
    tracking : bool, default False
        Whether the system has a tracker.

    Returns
    -------
    Series
        Boolean series with True for data on days that are 'sunny'
        according to the criteria above.

    """
    # ...

Two function API

def tracking_days(power_or_irradiance, daytime, correlation_min=0.94,
                  fixed_max=0.96, power_min=0.1):
    """Flag days where the data matches the profile of a tracking PV system.

    Tracking days are identified by fitting a restricted quartic to
    the data for each day. If the :math:`r^2` for the fit is greater
    than `correlation_min` and the :math:`r^2` for a quadratic fit is
    less than `fixed_max` then the data on that day is all marked
    True. Data on days where the tracker is stuck, or there is
    substantial variability in the data (i.e. caused by cloudiness) is
    marked False.

    Parameters
    ----------
    power_or_irradiance : Series
        Timezone localized series of power or irradiance measurements.
    daytime : Series
        Boolean series with True for times that are during the
        day. For best results this mask should exclude early morning
        and evening as well as night. Morning and evening may have
        problems with shadows that interfere with curve fitting, but
        do not necessarily indicate that the day was not sunny.
    correlation_min : float, default 0.94
        Minimum :math:`r^2` for a day to be considered sunny.
    fixed_max : float, default 0.96
        Maximum :math:`r^2` for a quadratic fit, if the quadratic fit
        is better than this, then tracking/fixed cannot be determined
        and the day is marked False.
    power_min : float, default 0.1
        Data less than `power_min * power_or_irradiance.mean()` is
        removed.

    Returns
    -------
    Series
        Boolean series with True for every value on a day that has a
        tracking profile (see criteria above).
    """
    # ...


def fixed_days(power_or_irradiance, daytime, correlation_min=0.94,
               power_min=0.4):
    """Flag days where the data matches the profile of a fixed PV system.

    Fixed days are identified when the :math:`r^2` for a quadratic fit
    to the power data is greater than `correlation_min`.

    Parameters
    ----------
    power_or_irradiance : Series
        Timezone localized series of power or irradiance measurements.
    daytime : Series
        Boolean series with True for times that are during the
        day. For best results this mask should exclude early morning
        and evening as well as night. Morning and evening may have
        problems with shadows that interfere with curve fitting, but
        do not necessarily indicate that the day was not sunny.
    correlation_min : float, default 0.94
        Minimum :math:`r^2` for a day to be considered sunny.
    power_min : float, default 0.1
        Data less than `power_min * power_or_irradiance.mean()` is
        removed.

    Returns
    -------
    Series
        True for values on days where `power_or_irradiance` matches
        the expected parabolic profile for a fixed PV system.

    """
    # ...

[cwhanse]

A design rule we've generally followed in pvlib is to implement the base layer as separate functions that stand alone. This approach has encouraged reusable functions. Applied here, I'd prefer two separate functions, one for fixed and one for tracked systems. We can add a function with the kwarg you propose as a wrapper on the two base functions. I think that the PVFleets workflow uses these sunny day identifiers to confirm, or discover, if a system is fixed or tracked. Separate base functions fit nicely in an if/then logic for that workflow.

[wfvining]

The original implementation from the PVFleets QA project includes a step to 'fail' the fit on days where the maximum power is less than the mean positive power (power[power > 0].mean()) for tracking systems. This makes it so the fit will only succeed on fairly sunny days and will fail on overcast days. The fixed function does not include this check. I need to decide whether to include this step in both or neither of the functions, but I don't think it should be in one and not the other.

My current thinking is to remove it since other functions (i.e. those in features.clearsky) can be combined with these to get sunny & tracking or sunny & fixed. (might need to change the title of this issue in that case).