Feature: revamp of wind speed-power curve filtering class

[Bartdoekemeijer]

This PR is not yet ready to be merged.

Feature or improvement description
This is a revamp of the windspeed-power curve filtering class. Specifically, it collapses steps 2-4 from the raw data processing workflow into a single processing step. It also provides a much more generic interface for filters and related insightful plots.

Related issue, if one exists
This resolves #67 but that's not the focus of this contribution.

Impacted areas of the software
The turbine_analysis subfolder.

Additional supporting information
The interface has a number of major changes:

• Most importantly: provide a generic interface for data filtering and plotting. For example, one can aggregate as many queries as one would like, and they will all be plotted in the same figure for insight, for example

    for ti in range(n_turbines):
        # Filter for numerical issues
        ws_pow_filtering.filter_by_condition(
            condition=(ws_pow_filtering.df["ws_{:03d}".format(ti)] < -1.0e-6),
            label="Wind speed below zero",
            ti=ti,
            verbose=True,
        )

        # Filter for numerical issues
        ws_pow_filtering.filter_by_condition(
            condition=(ws_pow_filtering.df["pow_{:03d}".format(ti)] < -1.0e-6),
            label="Power below zero",
            ti=ti,
            verbose=True,
        )

        ...

This does increase the length of a user's code, but I think it's particularly helpful and insightful. Users can be very particular in their queries, and as complex as they like. Each query is saved and linked to particular data indices in the dataset, which can then be plotted. See the test results below.

• Become independent from OpenOA for the time being, see Issue [BUG] Dependency conflict between FLASC and OpenOA #67
• Focus on turbine-individual level of filtering, not yet farm-level filtering. Hence, moving towards using for-loops on the user side. This also gives the user more precision, e.g., to filter with specific rules for specific turbines.
• Remove the filtering by standard deviation. This function became superfluous with the introduction of iterative filtering based on the estimated turbine power curve l;ast y
• Significantly reduce total lines of code and remove superfluous code.
• Improve the plot highlighting the identified filters over time, to highlight times of maintenance, turbine downtime, etc.

Test results, if applicable
Will open a pull request in the flasc_cookiecutter_template, but here's the relevant code snippet:

    # Apply a set of logic filters on the turbine measurements
    n_turbines = dfm.get_num_turbines(df)
    for ti in range(n_turbines):
        # Filter for NaN wind speed or power productions
        ws_pow_filtering.filter_by_condition(
            condition=(
            ws_pow_filtering.df["ws_{:03d}".format(ti)].isna() |
            ws_pow_filtering.df["pow_{:03d}".format(ti)].isna()
            ),
            label="Wind speed and/or power is NaN",
            ti=ti,
            verbose=True,
        )

        # Filter for numerical issues
        ws_pow_filtering.filter_by_condition(
            condition=(ws_pow_filtering.df["ws_{:03d}".format(ti)] < -1.0e-6),
            label="Wind speed below zero",
            ti=ti,
            verbose=True,
        )

        # Filter for numerical issues
        ws_pow_filtering.filter_by_condition(
            condition=(ws_pow_filtering.df["pow_{:03d}".format(ti)] < -1.0e-6),
            label="Power below zero",
            ti=ti,
            verbose=True,
        )

        # Filter for numerical issues
        ws_pow_filtering.filter_by_condition(
            condition=(ws_pow_filtering.df["ws_{:03d}".format(ti)] > 50),
            label="Wind speed above 50 m/s",
            ti=ti,
            verbose=True,
        )

        # Filter for numerical issues: note, make sure power is in kW
        ws_pow_filtering.filter_by_condition(
            condition=(ws_pow_filtering.df["pow_{:03d}".format(ti)] > 30e3),
            label="Power above 30 MW",
            ti=ti,
            verbose=True,
        )

        # Filter for power production is zero above cut-in wind speeds
        ws_pow_filtering.filter_by_condition(
            condition=(ws_pow_filtering.df["ws_{:03d}".format(ti)] > 4.0) & (ws_pow_filtering.df["pow_{:03d}".format(ti)] < 1.0),
            label="Power below 1 kW while wind speed above 4 m/s",
            ti=ti,
            verbose=True,
        )

        # Other common filters here are based on turbine-specific/OEM-specific flags. For example,
        # the 'run counter' in Vestas turbines indicates how many seconds of a 10-minute period
        # the turbine was reporting valid measurements. By filtering for the condition that
        # that value is 600, or above 590, you should be able to quickly identify a large part
        # of faulty measurements. In this case, we have an operational_status flag.
        ws_pow_filtering.filter_by_condition(
            condition=(ws_pow_filtering.df["is_operation_normal_{:03d}".format(ti)] == False),
            label="Self-flagged (is_operation_normal==False)",
            ti=ti,
            verbose=True,
        )

        # Filter for sensor-stuck faults
        ws_pow_filtering.filter_by_sensor_stuck_faults(
            columns=["wd_{:03d}".format(ti), "ws_{:03d}".format(ti)],
            ti=ti,
            n_consecutive_measurements=3,
            stddev_threshold=0.001,
            plot=False,
        )

        # Flag curtailment by marking measurements with a high wind speed but
        # lower power production as faulty.
        ws_pow_filtering.filter_by_condition(
            condition=(
                (ws_pow_filtering.df["ws_{:03d}".format(ti)] > 10.2) &
                (ws_pow_filtering.df["pow_{:03d}".format(ti)] < 3200.0)
            ),
            label="Curtailment: wind speed above 10.2 m/s but power below 3200 kW",
            ti=ti,
            verbose=True,
        )

        # Now filter iteratively by deviations from the median power curve
        ws_pow_filtering.filter_by_power_curve(
            ti=ti,
            ws_deadband=1.5,
            pow_deadband=70.0,
            cutoff_ws=20.0,
            m_pow_rb=0.97,
        )

        # Plot and save data for current dataframe
        ws_pow_filtering.plot_filters_in_ws_power_curve(ti=ti, fi=fi)
        ws_pow_filtering.plot_filters_in_time(ti=ti)
        ws_pow_filtering.plot_postprocessed_in_ws_power_curve(ti=ti, fi=fi)
        print("\n")

    # Plot farm-averaged power curve based on the data (useful to e.g., feed into FLORIS when not provided by OEM)
    ws_pow_filtering.plot_farm_mean_power_curve()

    # Get filtered dataframe and power curve
    df = ws_pow_filtering.get_df()
    df_pow_curve = ws_pow_filtering.pw_curve_df

And yields

Faulty measurements for WTG 002 increased from 0.015 % to 0.015 %. Reason: 'Wind speed and/or power is NaN'.
Faulty measurements for WTG 002 increased from 0.015 % to 0.315 %. Reason: 'Wind speed below zero'.
Faulty measurements for WTG 002 increased from 0.315 % to 0.315 %. Reason: 'Power below zero'.
Faulty measurements for WTG 002 increased from 0.315 % to 0.315 %. Reason: 'Wind speed above 50 m/s'.
Faulty measurements for WTG 002 increased from 0.315 % to 0.315 %. Reason: 'Power above 30 MW'.
Faulty measurements for WTG 002 increased from 0.315 % to 0.319 %. Reason: 'Power below 1 kW while wind speed above 4 m/s'.
Faulty measurements for WTG 002 increased from 0.319 % to 2.161 %. Reason: 'Self-flagged (is_operation_normal==False)'.
Faulty measurements for WTG 002 increased from 2.161 % to 2.178 %. Reason: 'Sensor-stuck fault'.
Faulty measurements for WTG 002 increased from 2.178 % to 5.283 %. Reason: 'Curtailment: wind speed above 10.2 m/s but power below 3200 kW'.
Faulty measurements for WTG 002 increased from 5.283 % to 5.365 %. Reason: 'Mean power curve outlier'.

and

To do before merging, in no particular order:

• Add simple unittests
• Remove superfluous code rather than just commenting out
• Expand docstrings and clean up comments generally
• Consider renaming ws_pow_filtering to turbine_filtering
• Prepare PR for flasc_cookiecutter_template

[Bartdoekemeijer]

See the pull request on flasc_cookiecutter_template for an example of the new functionalities.

[Bartdoekemeijer]

OK ready to be merged. @paulf81, @misi9170, could you take a look?

[paulf81]

I like it Bart, agree it's neat and readable! OK to merge from my end, thank you!

[misi9170]

@Bartdoekemeijer I agree, this looks really nice, sorry that I haven't put a comment in before now. I was hoping to get it tested using the flasc_cookiecutter_template, but still haven't quite figured out how to run things there based on your fork. If you'd like to get this in ASAP, I think you can go ahead and merge; otherwise, I'll hopefully be able to add a proper review once we've met next week!

[Bartdoekemeijer]

@misi9170 no worries, this can wait until next week! Happy to have your comments!

[misi9170]

@Bartdoekemeijer; all looks good to me, both here and in flasc_cookiecutter_template. I'm going to go ahead and approve and then merge both.