Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem with measurements spanning midnight UTC #27

Closed
dnorgaard-usgs opened this issue Dec 15, 2017 · 2 comments

Comments

Projects
None yet
1 participant
@dnorgaard-usgs
Copy link
Contributor

commented Dec 15, 2017

On Thu, Dec 14, 2017 at 5:07 PM, Kern, Christoph wrote:
Diana

I wonder if you could look into one other issue regarding MobileDOAS for us: Please see the below email from Tamar Elias. She and her group at HVO use MobileDOAS to make measurements at Kilauea. They are running into problems when collecting measurements that span a time window in which the date changes. This happens in Hawaii when using UTC, since they are about 12 hours off.

This brings up the larger question of: what is the correct way to set up the time? Up until now, I always thought it should not matter what the computer time is set to because the GPS should always be able to supply accurate time in UTC. This is the time that should be used to stamp the data.

However, I'm not sure whether it's used to create the data output directory. As you know, data is written to a different directory every day, and I think this directory creation uses the computer time. Well, we had some problems with assigning incorrect dates to datasets, so we decided to simply set the acquisition laptop time to UTC. Unfortunately, this is now causing other problems as per Tamar's email below.

Do you think you could have a look and see

  1. what time is used to create the output directory? (computer or GPS? UTC or local?)
  2. what time is used to label each traverse's folder? (is this the computer time?)
  3. what time is used to stamp each data point during acquisition (I assume this is GPS? in UTC?)
  4. what happens if a traverse spans into a new day? Can you reproduce Tamar's error?

Thanks so much for all your help with this!!!

Christoph

---------- Forwarded message ----------
From: Elias, Tamar
Date: Tue, Nov 28, 2017 at 9:20 PM
Subject: MDOAS date/time issue
To: Christoph Kern

hahahah - like this even should be on the radar at all....given all the other stuff. But maybe you could file this away and revisit it at some later date? Setting the computer time to utc creates 2 date folders for a days data - and a 'broken' traverse for the traverse that spans the 1400 HST hour. I would guess it might impact other users - even if they are not 10 hrs off utc....

Problem example:
We collected data on November 28 ~1400 HST. This would be November 29, 0000 UTC. The file labels the date time as November 28 0000 UTC. So the correct UTC time but incorrect UTC date. It is a hybrid HST/UTC date/time.

Could it be taking the date off of the original folder name? then correct for the UTC time but not the date? It's such slick software - just wondered if it might be worth fixing?🌋

Here is an example. Data collected 11/28:

mDOAS Emission Rate Result
COC_20171128_mDOAS33_20171128_1354evaluationLog_SO2
SpectrumNumberRange : 97 - 365
TraverseTime : 28-Nov-2017 00:04:11 UTC
TraverseAltitude : 0 m
WindSpeed : 5.9 +- 1 m/s (from UserInput)
WindDirection : 60 +- 4 deg (from VentLocation)
SO2EmissionRate : 161 +- 28 t/d
SO2EmissionRate : 1.86 +- 0.32 kg/s
Volcano : Kilauea-PuuO`o
VolcanoLatitude : 19.389 N
VolcanoLongitude : -155.107 E
VolcanoElevation : 720 m
VolcanoDistance : 11143 m
DataFilters : Baseline : -5.266e+15 molec/cm2; MinIntensity : N/A; MaxIntensity : N/A

mDOAS Emission Rate Result
COC_20171128_mDOAS34_20171128_1409evaluationLog_SO2
SpectrumNumberRange : 36 - 294
TraverseTime : 28-Nov-2017 00:16:58 UTC
TraverseAltitude : 0 m
WindSpeed : 5.9 +- 1 m/s (from UserInput)
WindDirection : 58 +- 3 deg (from VentLocation)
SO2EmissionRate : 604 +- 103 t/d
SO2EmissionRate : 6.99 +- 1.20 kg/s
Volcano : Kilauea-PuuO`o
VolcanoLatitude : 19.389 N
VolcanoLongitude : -155.107 E
VolcanoElevation : 720 m
VolcanoDistance : 11057 m
DataFilters : Baseline : 8.630e+15 molec/cm2; MinIntensity : N/A; MaxIntensity : N/A

@dnorgaard-usgs

This comment has been minimized.

Copy link
Contributor Author

commented Dec 15, 2017

On Fri, Dec 15, 2017 at 2:04 PM, Kern, Christoph wrote:
Diana

The GPS communicates with the NOVAC software using the so-called NMEA protocol. Basically, the GPS simply sends ASCII sentences to the virtual serial port every second. Each sentence starts with $GPXXX where XXX is a 3 character identifier that specifies the format of the rest of the sentence. The different sentence syntaxes are described in detail here:

http://www.gpsinformation.org/dale/nmea.htm

As you can see, the time is transmitted in almost every sentence. The date, however, is only transmitted in the $GPRMA sentence, so this is the one you would have to listen for for that. the format is DDMMYY. You might search the MobileDOAS code for $GPGGA or $GPRMA to find the spot where this takes place (?)

Let me know if I can help you out somehow...

Christoph

On Fri, Dec 15, 2017 at 10:52 AM, Norgaard, Diana wrote:
Hi,

From what I can tell, the output folders are using locatime, as is the date in the measurement file. Only the scan time uses the GPS time. Since Tamar mentioned using UTC creates two date folders and splits the traverse, I suspect the computer was set to localtime?

I should be able to use GPS time for the measurement date also, but I'm not quite sure yet what format the GPS date/time comes in. Here's how the code currently gets the hr/min/sec from GPS time if anyone sees the format from it:

void GetHrMinSec(int time, int &hr, int &min, int &sec){
hr = time/10000;
min = (time - hr*10000)/100;
sec = time % 100;

// make sure that there's no numbers larger than 60 (or 24) !!!
if(sec > 60){
	sec -= 60;
	min +=1;
}
if(min > 60){
	min -= 60;
	hr += 1;
}
hr = hr % 24;

}

I think I need an outdoor, beachfront table to work on in Hawaii so I can plug in the GPS and test the changes while I sip a cocktail listening to the crashing waves. I suppose the picnic table at cloudy CVO will have to do though....

Diana

On Fri, Dec 15, 2017 at 9:27 AM, Kern, Christoph wrote:
Diana and Tamar

Sorry for the somewhat confusing email last night. I did a little more investigating on this issue this morning and I think I found the problem. MobileDOAS does indeed use UTC time (I guess from the GPS?) to tag each spectrum as it is saving it. The date, however, must be taken from somewhere else and is not always correct.

I'm attaching 2 spectra from Tamar's data that were collected on Chain of Craters Road. Spectrum 95 was taken on 28 Nov 2017 23:59:59 UTC. Spectrum 96 was taken on 29 Nov 2017 00:00:02 UTC. However, spectrum 96 is tagged with the incorrect date (28 November). All subsequent spectra in this measurement also have the incorrect date. In fact, all subsequent measurements collected on this day are tagged with the wrong date, even after a new data collection sequence was started!

I don't quite understand this behaviour. My suspicion was that the date comes from the computer time, and so if the computer is not set to UTC, then the date may be wrong. But I think Tamar has set the computer time to UTC for this measurement - is this true Tamar? If so, that can't be the reason.

What is the desired behaviour here? Well, it seems clear to me that each spectrum should be tagged with the GPS date and time in UTC, regardless of what the computer time is set to. It may be best to use the computer time, however, to create directories for each measurement and tag the file and directory names using that time. This way each observatory can choose how they want to organize their data. If they want to use UTC, then they set their computer to UTC. If they prefer to save their data in directories according to their own local date, then they set the computer to local time. This has the advantage that measurements taken on the same day in rapid succession do not end up in different directories.

Does everyone agree with this strategy? I'm CCing Santiago on this email so he can weigh in.

Diana, whenever you have time, perhaps you can investigate how the spectra are currently being tagged (i.e. why the date is not incremented properly). The date is sent to the computer every second from the GPS, and since the program already reads latitude and longitude etc at this time interval, it should not really be a problem to also read the correct date and time.

Thank you all for your help sorting this stuff out!

Christoph

@dnorgaard-usgs

This comment has been minimized.

Copy link
Contributor Author

commented Dec 15, 2017

On Fri, Dec 15, 2017 at 2:23 PM, Kern, Christoph wrote:
yes, sorry $GPRMC has the date.

On Fri, Dec 15, 2017 at 2:22 PM, Norgaard, Diana wrote:
Thanks, this helps. Looks like time format is hhmmss.ss, which makes sense. Did you mean GPRMC? I don't see GPRMA. There is support in code for GPRMC and GPRMA, but need to modify it to read the date if GPRMC. I'll give it a shot.

@dnorgaard-usgs dnorgaard-usgs self-assigned this Dec 22, 2017

@dnorgaard-usgs dnorgaard-usgs added the bug label Jan 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.