Corrupted MagStar GMAG CDFs from cdf_make_vars

[jameswilburlewis]

From Cindy:

The pipeline to produce CDFs from MagStar GMAG data is producing corrupt CDFs.

Below is a summary of the MagStar procedures:

DOWNLOAD:
Download code includes:
/disks/socware/thmsoc_dp_current/src/ksh/magstar_get_csv.bash
/disks/socware/thmsoc_dp_current/src/python/magstar/dasi_themis_downloader.py

Download Process:
The bash script (cronjob) calls python code dasi_themis_downloader.py
dasi_themis_downloader downloads the last 21 days of csv files and copies them to the magstar ‘work2do’ directory /disks/themisdata/workdir/magstar2do/.

CONVERT TO CDF:
Conversion code includes:
/disks/socware/thmsoc_dp_current/src/ksh/magstar_csv2all.ksh
/disks/socware/thmsoc_dp_current/idl/thmsoc/magstar_gmag/thm_magstar_l2gen.pro

Conversion Process:

The ksh script (cronjob) magstar_csv2all.ksh lists the csv files that are in the magstar workdir /disks/themisdata/workdir/magstar2do and loops though each file calling the IDL routine thm_magstar_l2gen.pro
thm_magstar_l2gen.pro reads the csv file, uses the mastercdf file to create an l2_structure and calls cdf_save_vars to create the CDF file.

NOTES:

The Download process works well. Files have been downloaded through March 2024. The cronjob is currently commented out since the CDF files are not correct.
There is a file ready to be processed in the /disks/themisdata/workdir/magstar2do directory. All you need to do is run the script magstar_csv2all.ksh and it will find the file and create a CDF.
If you want to play with more csv files there is a temp directory disks/themisdata/workdir/magstar2do/temp/ that contains many csv files that can be copied up a directory and run.
A CDF file is created. It cannot, however, be loaded by thm_load_gmag. It is supposedly a corrupt CDF and is deleted when you try and load. Similarly, it cannot be loaded by matlab.
It occurred to me that perhaps the thm_load_gmag routine cannot handle the new magstar CDF files. They are using TT2000 datetime which is different from the time/range/epoch used by the older gmag files. Or possibly the CDF version is not current. I'll look into that.

[jameswilburlewis]

Cindy said today that the cdf_save_vars code may be producing readable CDFs on her Mac, so that's another hint that the issue might have more to do with the environment or IDL installation than with cdf_save_vars itself.

[jimm02]

So the problem is with the mastercdfs for the files that aren't working. The bad masters have the Magnetic field variable as non-record varying, and this causes the data_dimen variable to be (n,3) rather than 3, causing the crash.

[jimm02]

Fixed, what was hapopning is that the l2gen program expected the time variable as the first tag in the structure loaded from the mastercdf, and the B field as the second. For the 'bad' files this was not the case, the first variable was the Bfield, the seond variable was units, and the time variable was fourth or fifth. The code that tests for this (see, e.g. thm_maccs_gmag_l2gen) was commented out in the magstar l2gen file. I fixed this, and updated the production IDL code. It should work now.

[jimm02]

I reprocessed all of the files for 2024-03-19, they can be loaded, but the times are screwy.

[jimm02]

Ok, I fixed the times in the magstar files. The problem was that the time array was defined as CDF_REAL8, and needed to be CDF_TIME_TT2000. This could be fixed in the mastercdfs for dmo, dhe, dbo, doh, dat, dma, and dme, but easier to add a line in the l2gen program that sets the datatype. I re-reprocessed all of the files for 2024-03-19, and no more problems.

[clrussell90404]

Yay. Thanks Jim! I appreciate your help on this! I will check and run the code and then reprocess.

…

________________________________ From: James McTiernan ***@***.***> Sent: Wednesday, April 24, 2024 11:13 AM To: spedas/bleeding_edge ***@***.***> Cc: Russell, Cindy ***@***.***>; Assign ***@***.***> Subject: Re: [spedas/bleeding_edge] Corrupted MagStar GMAG CDFs from cdf_make_vars (Issue #154) Ok, I fixed the times in the magstar files. The problem was that the time array was defined as CDF_REAL8, and needed to be CDF_TIME_TT2000. This could be fixed in the mastercdfs for dmo, dhe, dbo, doh, dat, dma, and dme, but easier to add a line in the l2gen program that sets the datatype. I re-reprocessed all of the files for 2024-03-19, and no more problems. — Reply to this email directly, view it on GitHub<#154 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A5NNFAXRC5DPTP6PWR5IWFTY67R45AVCNFSM6AAAAABGTP2OMKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZVGQ2DQMRXGU>. You are receiving this because you were assigned.Message ID: ***@***.***>