Theilsenplot: Code optimization

[Tanvi-Jain01]

@nipunbatra , @patel-zeel

Code Optimization:

Removing unnecessary import statements

vayu/vayu/TheilSen.py

Lines 25 to 28 in ef99aef

	import seaborn as sns
	import scipy
	from scipy import stats
	import math

Original Code:

vayu/vayu/TheilSen.py

Lines 46 to 63 in ef99aef

	i = 0
	x = 0
	j = 0
	var2 = []
	while i < num_unique_years:
	df_new = df[year[j]].resample("1D").mean()
	df_new = df_new.fillna(method="ffill")
	df_new["month"] = df_new.index.month
	# df_new['day']=df_new.index.dayofweek
	# df_new['hour']=df_new.index.hour
	i = i + 1
	j = j + 1
	x = 0
	while x < 12:
	a = df_new[df_new.month == x]
	mean_var2 = a[pollutant].mean()
	var2.append(mean_var2)
	x = x + 1

Optimized code:

for year in unique_years:
    df_year = df[df.index.year == year]
    df_monthly = df_year.resample("M").mean()
    monthly_mean = df_monthly[pollutant].values
    var2.extend(monthly_mean)
    scatterX.extend(np.arange(len(monthly_mean)) / 12 + year)

Explanation:
In the original code, there are nested loops where the outer loop iterates over num_unique_years and the inner loop iterates over 12 months. Inside the inner loop, a new DataFrame df_new is created by resampling the data with a daily frequency ("1D") and then filling missing values with forward fill. The monthly mean of the pollutant is calculated for each month, and the values are appended to the var2 list.

Resampling Improvement: In the optimized code, the need for nested loops is eliminated. Instead, a single loop iterates over unique_years. For each year, a new DataFrame df_year is created by filtering the data based on the year. Then, the resample method with a frequency of "M" (monthly) is directly applied to df_year, which returns a new DataFrame df_monthly with monthly mean values for the pollutant. The monthly mean values are extracted using df_monthly[pollutant].values and then extended to the var2 list.

Original Code:

vayu/vayu/TheilSen.py

Lines 37 to 42 in ef99aef

	i = 0
	year = []
	while i < len(unique_years):
	year.append(str(unique_years[i]))
	i = i + 1
	num_unique_years = len(year)

Optimized Code:

year = [str(year) for year in unique_years]
num_unique_years = len(year)

Explanation:
List Comprehension: In the optimized code, a list comprehension is used to achieve the same result more concisely. The list comprehension iterates over unique_years and converts each year to a string using the expression str(year). The resulting strings are directly added to the year list. This eliminates the need for an explicit loop and simplifies the code.

Improved Code:

def TheilSen(df, pollutant):
    import matplotlib.pyplot as plt
    import pandas as pd
    import numpy as np
    from matplotlib.ticker import ScalarFormatter

    df.index = pd.to_datetime(df.date)
    unique_years = np.unique(df.index.year)

    var2 = []
    scatterX = []

    for year in unique_years:
        df_year = df[df.index.year == year]
        df_monthly = df_year.resample("M").mean()
        monthly_mean = df_monthly[pollutant].values
        var2.extend(monthly_mean)
        scatterX.extend(np.arange(len(monthly_mean)) / 12 + year)

    y = np.array(var2)
    x = np.array(scatterX)

    def best_fit(X, Y):
        xbar = np.mean(X)
        ybar = np.mean(Y)
        n = len(X)

        numer = np.sum(X * Y) - n * xbar * ybar
        denum = np.sum(X ** 2) - n * xbar ** 2

        b = numer / denum
        a = ybar - b * xbar

        return a, b

    a, b = best_fit(x, y)

    fig, ax = plt.subplots()
    ax.plot(x, y, "-o")
    ax.set_xlabel("Year")
    ax.set_ylabel(pollutant)
    ax.set_title("TheilSen plot")
    plt.plot(np.unique(x), np.poly1d(np.polyfit(x, y, 1))(np.unique(x)), color="red")

    # Format y-axis tick labels
    ax.xaxis.set_major_formatter(ScalarFormatter(useOffset=False))
    plt.savefig("TheilSenplot.png", bbox_inches="tight")
    print("Your plot has also been saved")

    plt.show()

TheilSen(df1, 'pm25')

Output: