SDL data contains CP1 data

[jasonebox]

as late as 2022-02-11 17:00:00, SDL hourly and daily data from Thredds contains what I think is CP1 data, at least the GPS data. I think the issue is a mismatch with IMEI

using code provided below, I get this plot:

names=['SDL']
site='SDL'
timeframe='hour'
for i,name in enumerate(names):
    if i>=0:

        print()
        print(i,n_sites-i,name)#[i],meta.name[i],names[i][0:5])
        
        site=name

        # fn=aws_data_path+site+'/'+site+'_day.csv'
        # df=pd.read_csv(fn)        
        # print(site)
        url = "https://thredds.geus.dk/thredds/fileServer/aws_l3_station_csv/level_3/{}/{}_{}.csv".format(site,site,timeframe)
        df = pd.read_csv(url)
        print(df)
        
        print(df.columns)

        
        n=20
        lat=np.nanmean(df.gps_lat)#[-n:])
        lon=-abs(np.nanmean(df.gps_lon))#[-n:])
        elev=np.nanmean(df.gps_alt)
        # print(lat,lon,elev)
        
        # df.index = pd.to_datetime(df.time)
        # fig, ax = plt.subplots(figsize=(10,10))
        # plt.plot(df.batt_v)
        # plt.ylabel('VDC')
        # plt.title(name+' 2023 from Thredds')
        # plt.setp(ax.xaxis.get_majorticklabels(), rotation=90,ha='center' )

        df["date"] = pd.to_datetime(df.time)

        df['year'] = pd.DatetimeIndex(df["date"]).year
        df['day'] = pd.DatetimeIndex(df["date"]).day
        df['hour'] = pd.DatetimeIndex(df["date"]).hour
        df['month'] = pd.DatetimeIndex(df["date"]).month
        df['doy'] = pd.DatetimeIndex(df["date"]).dayofyear
        df['doy_dec'] = df['doy']+(df['hour']-1)/23
        df.index = pd.to_datetime(df.time)
    
        print(df.columns)
        # print(df)
        

        # lat=df.gps_lat
        # lon=df.gps_lon
        # elev=df.gps_alt
#%%
        plt.close()
        fig, ax = plt.subplots(3,1,figsize=(8,12))

        cc=0
        ax[cc].plot(df['date'],df['gps_lat'],'.',color='k')
        ax[0].set_title(name+' latitude')
        ax[cc].set_xticklabels([])

        cc+=1
        ax[cc].plot(df['date'],df['gps_lon'],'.',color='k')
        ax[cc].set_title(name+' longitude')
        ax[cc].set_xticklabels([])

        cc+=1
        ax[cc].plot(df['date'],df['gps_alt'],'.',color='k')
        ax[cc].set_title(name+' elevation')
        plt.setp(ax[cc].xaxis.get_majorticklabels(), rotation=90,ha='center' )
        
        
        x=df['date'][df['gps_lat']>69]
        print('last date with CP1 latitude',x[-1])

[BaptisteVandecrux]

Probably a problem of transmission file that did not get cropped properly:
GEUS-Glaciology-and-Climate/pypromice#179
#60

However I did not see any problem when uploading the AWS data to the dataverse on August 17, after the GC-Net fieldwork:
https://github.com/GEUS-Glaciology-and-Climate/PROMICE-AWS-diagnostic/blob/main/figures/new_dataverse_upload/SDL_10.png

[BaptisteVandecrux]

Slight update: It is not the transmission file. It comes from the logger files:

The 2023 logger file contains CP1 data before 2022-06-23 14:40:00

This was not reflected in the update of the SDL toml file that asks pypromice to process the entire 2023 logger file.

So the SDL raw toml file should be updated to use the 2023 file after 2022-06-23 14:40:00 and the 2022_corrected before that.

@robertfausto you authored the SDL toml file last. Will you make the changes?

[robertfausto]

CP1 lines in files removed

[BaptisteVandecrux]

The data is now removed successfully from the THREDDS server, but remains on the dataverse.
See illustration below where "old" is what is currently on the dataverse and "new" is the current l3 data:

@PennyHow or @ladsmund can you trigger a dataverse update so we can mark this issue as solved?

[BaptisteVandecrux]

This fix was released on the dataverse V11 on Nov. 6.