Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix time-related columns for Obspack time series #147

Open
ZogopZ opened this issue Jun 12, 2023 · 1 comment
Open

Fix time-related columns for Obspack time series #147

ZogopZ opened this issue Jun 12, 2023 · 1 comment
Labels
bug Something isn't working

Comments

@ZogopZ
Copy link
Member

ZogopZ commented Jun 12, 2023

Currently obtaining data for an Obspack time series will return a pandas dataframe with a time column.
Example code:

from icoscp.cpb.dobj import Dobj
dobj = Dobj('https://meta.icos-cp.eu/objects/VvC1QkMAWW5Z9iUSFaPijY5W')
dobj.data

There are other time-related columns that should be in the dataframe but are not.
Here's the code that does the conversions:

"""
The ICOS Carbon Portal provides a TIMESTAMP which is
a unix timestamp in milliseconds [UTC]
Convert the "number" to a pandas date/time object
"""
if 'TIMESTAMP' in df.columns and self._dtconvert:
df['TIMESTAMP'] = pd.to_datetime(df.loc[:,'TIMESTAMP'],unit='ms')
if 'TIMESTAMP_END' in df.columns and self._dtconvert:
df['TIMESTAMP_END'] = pd.to_datetime(df.loc[:,'TIMESTAMP_END'],unit='ms')
# there are files with date and time where
# date is a unixtimestamp
# time seconds per day
if 'date' in df.columns and self._dtconvert:
df['date'] = pd.to_datetime(df.loc[:,'date'],unit='D')
if 'time' in df.columns and self._dtconvert:
# convert unix timestamp
df['time'] = pd.to_datetime(df.loc[:,'time'],unit='s')
# remove the data, so that only time remains
df['time'] = pd.to_datetime(df.loc[:,'time'],format='%H:%M').dt.time

@ZogopZ ZogopZ added the bug Something isn't working label Jun 12, 2023
@claudiodonofrio
Copy link
Member

as a temporary workaround, one can use the following code for obspack data, resulting in the correct date/time conversion:

from icoscp.cpb.dobj import Dobj
d = 'https://meta.icos-cp.eu/objects/Hk6TA59l1gIc0GQbmorv69A_'
do = Dobj(d)
do.dateTimeConvert = False
df
icos_LTR icos_SMR icos_STTB icos_datalevel qc_flag time value value_std_dev
NaN NaN NaN -2147483648 O 2006-01-06 08:30:00 0.000390 NaN
NaN NaN NaN -2147483648 O 2006-01-06 09:30:00 0.000390 NaN

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants