Issue with reading data dated to 31 Mar from IAGA-2002 files (Mac OS)

[teraita]

When reading the data to the 1min values there is issue with handling of the data of 31 March. The file is not read correctly between 03-05UT. When manipulating the date information in filename, and data row (detestr and doy) in IAGA-2002 format to respond 31 Jan the file works. Might be some issue with epoch/ stream lengths?

Problem sees to hit only to period 03-05UT:

Same data file, date information manipulated:

When merging XYZ and F of the date and calculating deltaF the values are corrupted, so it this not only visualisation problem in MagPy.

Is this only Mac related issue?

Please, request data samples, if needed.

[leonro]

Hi Tero, can you please send me the data files? I will check what going on.

[teraita]

Hi Roman, Here is the 1min SOD file with XYZ. Cheers, Tero Ps. We still have “chance” to get trough the group stage by stealing point from Belgium. At least about 3 hours, when Lukaku strikes… ;-)

…

On 21. Jun 2021, at 19.26, Roman Leonhardt ***@***.***> wrote: Hi Tero, can you please send me the data files? I will check what going on. Sorry for not replying earlier, but my notebook broke down and I needed some time to set up everything again... Best wishes, Roman Dr. Roman Leonhardt Conrad Observatorium ZAMG - Zentralanstalt für Meteorologie und Geodynamik 1190 Wien, Hohe Warte 38 Tel.: +43 1 36026 2507 Fax: +43 1 36866 21 E-Mail: ***@***.***<https://webmail.zamg.ac.at/owa/redir.aspx?SURL=t9AS8jCcPggsx8oPf2D18nQbebRT-hDvZYGVU9uNLKVg80_NrjnSCG0AYQBpAGwAdABvADoAcgBvAG0AYQBuAC4AbABlAG8AbgBoAGEAcgBkAHQAQAB6AGEAbQBnAC4AYQBjAC4AYQB0AA..&URL=mailto%3aroman.leonhardt%40zamg.ac.at> www.zamg.ac.at<https://webmail.zamg.ac.at/owa/redir.aspx?SURL=DgfuyOdHyWS5QXRRIyzOHmne5_jxCaCmQXd1H4B7AnJg80_NrjnSCGgAdAB0AHAAOgAvAC8AdwB3AHcALgB6AGEAbQBnAC4AYQBjAC4AYQB0AC8A&URL=http%3a%2f%2fwww.zamg.ac.at%2f> Conrad Observatorium 2763 Muggendorf , Trafelberg 1-2 Tel.: +43 2632 73276 oder 7367221 www.conrad-observatory.at<https://webmail.zamg.ac.at/owa/redir.aspx?SURL=3GEn2GkM89BDePjqOqAYT0d3pi9ZpUlcz8Zli53n9uNg80_NrjnSCGgAdAB0AHAAOgAvAC8AdwB3AHcALgBjAG8AbgByAGEAZAAtAG8AYgBzAGUAcgB2AGEAdABvAHIAeQAuAGEAdAAvAA..&URL=http%3a%2f%2fwww.conrad-observatory.at%2f> ________________________________ Von: Tero Raita ***@***.***> Gesendet: Donnerstag, 20. Mai 2021 14:15:28 An: geomagpy/magpy Cc: Subscribed Betreff: [geomagpy/magpy] Issue with reading data dated to 31 Mar from IAGA-2002 files (Mac OS) (#103) When reading the data to the 1min values there is issue with handling of the data of 31 March. The file is not read correctly between 03-05UT. When manipulating the date information in filename, and data row (detestr and doy) in IAGA-2002 format to respond 31 Jan the file works. Might be some issue with epoch/ stream lengths? Problem sees to hit only to period 03-05UT: [SOD20190331_03-05UT]<https://user-images.githubusercontent.com/22149196/118975443-b0e7da00-b97c-11eb-8cda-9aa3c9af40f9.png> [SOD20190331]<https://user-images.githubusercontent.com/22149196/118975410-a9283580-b97c-11eb-8229-ddf25757a31c.png> Same data file, date information manipulated: [SOD20190131]<https://user-images.githubusercontent.com/22149196/118975462-b6452480-b97c-11eb-883c-dcaa11e17398.png> When merging XYZ and F of the date and calculating deltaF the values are corrupted, so it this not only visualisation problem in MagPy. Is this only Mac related issue? Please, request data samples, if needed. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#103>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AD7YDSL7DIYWKM7LPESA643TOT4OBANCNFSM45G35ZEA>. EXTERNAL EMAIL: Do not click any links or open any attachments unless you trust the sender and know the content is safe. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#103 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFI7QTDMLP2Y2WDTNMHSTNTTT5R3BANCNFSM45G35ZEA>.

[teraita]

Hi, 2Hz IAGA-2002 files of 30-31Mar 2019. Cheers, Tero

…

On 21. Jun 2021, at 19.32, Tero Raita ***@***.***> wrote: Hi Roman, Here is the 1min SOD file with XYZ. Cheers, Tero Ps. We still have “chance” to get trough the group stage by stealing point from Belgium. At least about 3 hours, when Lukaku strikes… ;-) <sod20190331vmin.min> > On 21. Jun 2021, at 19.26, Roman Leonhardt ***@***.*** ***@***.***>> wrote: > > > Hi Tero, > > > can you please send me the data files? I will check what going on. Sorry for not replying earlier, but my notebook broke down and I needed some time to set up everything again... > > > Best wishes, > Roman > > > Dr. Roman Leonhardt > Conrad Observatorium > ZAMG - Zentralanstalt für Meteorologie und Geodynamik > 1190 Wien, Hohe Warte 38 > Tel.: +43 1 36026 2507 > Fax: +43 1 36866 21 > E-Mail: ***@***.***<https://webmail.zamg.ac.at/owa/redir.aspx?SURL=t9AS8jCcPggsx8oPf2D18nQbebRT-hDvZYGVU9uNLKVg80_NrjnSCG0AYQBpAGwAdABvADoAcgBvAG0AYQBuAC4AbABlAG8AbgBoAGEAcgBkAHQAQAB6AGEAbQBnAC4AYQBjAC4AYQB0AA..&URL=mailto%3aroman.leonhardt%40zamg.ac.at <https://webmail.zamg.ac.at/owa/redir.aspx?SURL=t9AS8jCcPggsx8oPf2D18nQbebRT-hDvZYGVU9uNLKVg80_NrjnSCG0AYQBpAGwAdABvADoAcgBvAG0AYQBuAC4AbABlAG8AbgBoAGEAcgBkAHQAQAB6AGEAbQBnAC4AYQBjAC4AYQB0AA..&URL=mailto%3aroman.leonhardt%40zamg.ac.at>> > www.zamg.ac.at <http://www.zamg.ac.at/><https://webmail.zamg.ac.at/owa/redir.aspx?SURL=DgfuyOdHyWS5QXRRIyzOHmne5_jxCaCmQXd1H4B7AnJg80_NrjnSCGgAdAB0AHAAOgAvAC8AdwB3AHcALgB6AGEAbQBnAC4AYQBjAC4AYQB0AC8A&URL=http%3a%2f%2fwww.zamg.ac.at%2f <https://webmail.zamg.ac.at/owa/redir.aspx?SURL=DgfuyOdHyWS5QXRRIyzOHmne5_jxCaCmQXd1H4B7AnJg80_NrjnSCGgAdAB0AHAAOgAvAC8AdwB3AHcALgB6AGEAbQBnAC4AYQBjAC4AYQB0AC8A&URL=http%3a%2f%2fwww.zamg.ac.at%2f>> > > > Conrad Observatorium > 2763 Muggendorf , Trafelberg 1-2 > Tel.: +43 2632 73276 oder 7367221 > www.conrad-observatory.at <http://www.conrad-observatory.at/><https://webmail.zamg.ac.at/owa/redir.aspx?SURL=3GEn2GkM89BDePjqOqAYT0d3pi9ZpUlcz8Zli53n9uNg80_NrjnSCGgAdAB0AHAAOgAvAC8AdwB3AHcALgBjAG8AbgByAGEAZAAtAG8AYgBzAGUAcgB2AGEAdABvAHIAeQAuAGEAdAAvAA..&URL=http%3a%2f%2fwww.conrad-observatory.at%2f <https://webmail.zamg.ac.at/owa/redir.aspx?SURL=3GEn2GkM89BDePjqOqAYT0d3pi9ZpUlcz8Zli53n9uNg80_NrjnSCGgAdAB0AHAAOgAvAC8AdwB3AHcALgBjAG8AbgByAGEAZAAtAG8AYgBzAGUAcgB2AGEAdABvAHIAeQAuAGEAdAAvAA..&URL=http%3a%2f%2fwww.conrad-observatory.at%2f>> > ________________________________ > Von: Tero Raita ***@***.***> > Gesendet: Donnerstag, 20. Mai 2021 14:15:28 > An: geomagpy/magpy > Cc: Subscribed > Betreff: [geomagpy/magpy] Issue with reading data dated to 31 Mar from IAGA-2002 files (Mac OS) (#103) > > > When reading the data to the 1min values there is issue with handling of the data of 31 March. The file is not read correctly between 03-05UT. When manipulating the date information in filename, and data row (detestr and doy) in IAGA-2002 format to respond 31 Jan the file works. Might be some issue with epoch/ stream lengths? > > Problem sees to hit only to period 03-05UT: > [SOD20190331_03-05UT]<https://user-images.githubusercontent.com/22149196/118975443-b0e7da00-b97c-11eb-8cda-9aa3c9af40f9.png <https://user-images.githubusercontent.com/22149196/118975443-b0e7da00-b97c-11eb-8cda-9aa3c9af40f9.png>> > > [SOD20190331]<https://user-images.githubusercontent.com/22149196/118975410-a9283580-b97c-11eb-8229-ddf25757a31c.png <https://user-images.githubusercontent.com/22149196/118975410-a9283580-b97c-11eb-8229-ddf25757a31c.png>> > > Same data file, date information manipulated: > [SOD20190131]<https://user-images.githubusercontent.com/22149196/118975462-b6452480-b97c-11eb-883c-dcaa11e17398.png <https://user-images.githubusercontent.com/22149196/118975462-b6452480-b97c-11eb-883c-dcaa11e17398.png>> > > When merging XYZ and F of the date and calculating deltaF the values are corrupted, so it this not only visualisation problem in MagPy. > > Is this only Mac related issue? > > Please, request data samples, if needed. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub<#103 <#103>>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AD7YDSL7DIYWKM7LPESA643TOT4OBANCNFSM45G35ZEA <https://github.com/notifications/unsubscribe-auth/AD7YDSL7DIYWKM7LPESA643TOT4OBANCNFSM45G35ZEA>>. > > EXTERNAL EMAIL: Do not click any links or open any attachments unless you trust the sender and know the content is safe. > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub <#103 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFI7QTDMLP2Y2WDTNMHSTNTTT5R3BANCNFSM45G35ZEA>. >

[leonro]

Well.... something I dont really understand: I am testing for leap seconds within the method LeapSeconds of stream.py. I am using the following conversion which actually should work fine in order to get a datetime compatible output:

t = "2019-03-31T02:00:00"
ti = time.strptime(t, "%Y-%m-%dT%H:%M:%S")
tim = datetime.fromtimestamp(time.mktime(ti))

tim, however, then results in 2019-03-31T03:00:00 which is obviously wrong. Wrong results are apparently only obtained on March 31, 2019 between 2:00 and 3:00.

Workaround:
tim = datetime(*ti[:5]+(min(ti[5], 59),))

[leonro]

The workaround does the job. Issue solved in commit: e6e5258