2 Issues: Getting data from the future and integrity of the datasets

[cminton-hub]

1. ISSUE: Getting data from dates in the future.

Steps to reproduce:
While using the export function on the website for all historically hourly data for station ID 10488 in Dresden (atm till tomorrow). Same while using python (future dates till 2020-08-21)

"""define the periode for fetching"""
start = datetime(2021, 8, 1)
end = datetime(2021, 8, 31, 23, 59)
station = "10488"

"""Get hourly data"""
df_weatherdata = Hourly(station, start, end)
df_weatherdata = df_weatherdata.fetch()
df_weatherdata.reset_index(inplace=True)
df_weatherdata.rename(columns={"time": "datetime"}, inplace=True)
df_weatherdata=df_weatherdata.assign(id_national=station)
print(df_weatherdata)

Why can i query future data???

2. ISSUE: Integrity
There might be a problem with the quality of the data.

Download the historically hourly data for station ID 10488 in Dresden and search for: 2021-03-01 00:00:00
RESULT:
2020-03-01 00:00:00,8.0,2.0,66.0,0.0,0.0,230.0,25.9,44.0,998.9,0.0,4.0,01048

Steps to reproduce:
"""define the periode for fetching"""
start = datetime(2020, 3, 1)
end = datetime(2020, 5, 31, 23, 59)
station = "10488"

"""Get hourly data"""
df_weatherdata = Hourly(station, start, end)
df_weatherdata = df_weatherdata.fetch()
df_weatherdata.reset_index(inplace=True)
df_weatherdata.rename(columns={"time": "datetime"}, inplace=True)
df_weatherdata=df_weatherdata.assign(id_national=station)
print(df_weatherdata)

RESULT:
datetime,temp,dwpt,rhum,prcp,snow,wdir,wspd,wpgt,pres,tsun,coco,id_national
2020-03-01 00:00:00,7.7,1.3,64.0,0.0,0.0,220.0,27.4,44.0,998.9,0.0,4.0,10488
Compare the data from both datasets -> they differ completely

[clampr]

I understand the confusion and I will add an FAQ page on the dev site soon to explain some of these common questions. Regrading the first issue: this is caused by the fact, that model=True by default. At Meteostat we use (MOSMIX) model data to fill gaps in the historical time series. As we do have these forecasts anyway, they're provided along with the historical data. If you don't want model data, you can opt-out on both the website and the Python lib.

Issue 2 is a bit more special and has to do with an order/priority fix in the latest version of the bulk data interface. We now choose SYNOP data over METAR reports because they're more accurate. This will be fixed in the next release of the web app. Until then, I'd prefer the Python results.

[cminton-hub]

@clampr Thanks for your answer.

1. Issue:
   As far as it conserns my usecase. I support the fact model=True by default for historically data while you can still opt-out. It will generate more uniform data. But still including the forecast into historically data is a bad idea and needs to be differentiated. There might be a better implementation splitting historical and forcast data. A second button for forcast data on the website might a solution (fetching all forcast data for a station/ region), for python maybe a forcast class?

2. Issue:
   Python is my prefered choice anyway. But the integrity is still given?

[clampr]

Well, the problem is that we rely on static station dumps. Data isn't queried directly from the Meteostat DB when you make a request through the Python lib. This approach allows the library to scale with minimal to no restrictions.

I agree that mixing historical data and forecasts is confusing. Also, I think there are better alternatives when it comes to weather forecasts. If we want to keep the ability of getting data for the current day, which I think is relevant to many users, we need to have at least a short-term forecast (48 hours) in the station dumps (because of update frequency and caching). For now, I limited the forecasts to 48 hours for all dumps. I think it's fair to say that Meteostat has a focus on historical data with short term forecasts for more comfortable analysis. As 48 hour forecasts are very accurate most of the time, I think it's okay to use them for analysis in certain situations.

If you're only using the Python interface, integrity should be given.

[cminton-hub]

Ok this is a good limitation.

If you're only using the Python interface, integrity should be given.

Perfect thanks.

An other question:

When comparing the output and just changing model=True to False I expect to get less rows/ or attributes with NaN/null values.
Steps to reproduce:

"""define the periode for fetching"""
start = datetime(2020, 3, 1)
end = datetime(2020, 5, 31, 23, 59)
station = "10488"

"""Get hourly data and opt-out for model data"""
df_weatherdata = Hourly(station, start, end, model=False)
df_weatherdata = df_weatherdata.fetch()
df_weatherdata.reset_index(inplace=True)
df_weatherdata.rename(columns={"time": "datetime"}, inplace=True)
df_weatherdata=df_weatherdata.assign(id_national=station)
print(df_weatherdata)

Results: While model=False I still get values or other values while model=True. A model might be even used when data is present?

For comparising:
10488_2020-03-01_2020-05-31_True.csv
10488_2020-03-01_2020-05-31_False.csv

compared csv files online:
https://www.textcompare.org/csv/compare?id=61165041052c9b0015ebb514

[clampr]

New docs on data quality. https://dev.meteostat.net/quality.html

[cminton-hub]

@clampr Thanks for all your work.

Steps I did afterwards:

1. I used python lib to get some sample data for stations in Dresden.
2. Get the data from dwd within the same timeframe
3. Compare both datasets --> They are identical --> Proof: routine fix works correctly for stations in Dresden, should probably work for all.