Problematic ACORN DM files

[ocehugo]

The following ACORN DM files are errors and we don't have them yet:

date	time	filename
2020-03-28	09:14:06.024959630	IMOS_ACORN_RV_20180105T144000Z_CWI_FV01_radial.nc
2020-03-27	19:38:31.132059356	IMOS_ACORN_RV_20180914T154500Z_CSP_FV01_radial.nc
2020-04-15	18:10:10.747370215	IMOS_ACORN_RV_20190111T095500Z_FRE_FV01_radial.nc
2019-12-11	22:00:08.720899711	IMOS_ACORN_RV_20191014T020000Z_DONG_FV01_radial.nc

The first file (CWI) appears OK, but the rest are failing at db ingestion/talend (date related). @lbesnard - any ideas?

The file below failed because of an incomplete (or no) global attributes. The file is currently published. @scosoli

date	time	filename
2019-06-04	14:48:36.120441805	IMOS_ACORN_V_20090327T090000Z_TURQ_FV00_1-hour-avg.nc

The files below failed for the same reason above, but they are not published. @scosoli

date	time	filename
2019-09-26	16:20:13.654942707	IMOS_ACORN_V_20091116T060000Z_TURQ_FV01_1-hour-avg.nc
2019-10-07	15:06:57.929240358	IMOS_ACORN_V_20091116T060000Z_TURQ_FV01_1-hour-avg.nc

[scosoli]

@yashahoo can you check CWI CSP FRE DONG FV01_radials? I will check TURQ data

[yashahoo]

i will look into it.

…

On 3 Jun 2020, at 3:49 pm, scosoli ***@***.***> wrote: @yashahoo <https://github.com/yashahoo> can you check CWI CSP FRE DONG FV01_radials? I will check TURQ data — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#459 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIA7YXUDXBKX4SRAVKALDSDRUX57BANCNFSM4NPTRIOA>.

[scosoli]

@ocehugo, I don't see how IMOS_ACORN_V_20090327T090000Z_TURQ_FV00_1-hour-avg.nc has global attributes different from or missing in comparison to other files. Say, compare for instance with IMOS_ACORN_V_20090327T100000Z_TURQ_FV00_1-hour-avg.nc or IMOS_ACORN_V_20090327T110000Z_TURQ_FV00_1-hour-avg.nc.
can you provide more details?

[scosoli]

if I ncdump -h the suspect and the following file, the only differences I get are the following:
A diff between the two ncdump of the two files reports:

---

< title: IMOS ACORN Turqoise Coast (TURQ), one hour averaged current RT-QC data, 2009-03-27T09:00:00Z\

title: IMOS ACORN Turqoise Coast (TURQ), one hour averaged current RT-QC data, 2009-03-27T10:00:00Z
15,16c15,16
< date_created: 2019-06-04T09:38:57Z
< abstract: Sea State data, CODAR Ocean Sensors. SeaSonde sea surface radar located at Turquoise Coast (WA), for time 2009-03-27T09:00:00Z. Sea State represents the surface sea water state obtained by merging the radial surface sea water state components from two or more radar stations. Data produced by Australian Coastal Ocean Radar Network, Integrated Marine Observing System.\

---

date_created: 2019-06-04T09:38:59Z
abstract: Sea State data, CODAR Ocean Sensors. SeaSonde sea surface radar located at Turquoise Coast (WA), for time 2009-03-27T10:00:00Z. Sea State represents the surface sea water state obtained by merging the radial surface sea water state components from two or more radar stations. Data produced by Australian Coastal Ocean Radar Network, Integrated Marine Observing System.
33,34c33,34
< time_coverage_start: 2009-03-27T09:00:00Z
< time_coverage_end: 2009-03-27T09:00:00Z\

---

time_coverage_start: 2009-03-27T10:00:00Z
time_coverage_end: 2009-03-27T10:00:00Z\

that is, differences between global attributes is in the creation time stamps, and in the time coverage. I do not see any major differences other than those -expected- reported above

[yashahoo]

@ocehugo
I don't see a problem with IMOS_ACORN_RV_20180105T144000Z_CWI_FV01_radial.nc either and have re-uploaded the file I have locally.
These 2 files :
IMOS_ACORN_RV_20180914T154500Z_CSP_FV01_radial.nc
IMOS_ACORN_RV_20190111T095500Z_FRE_FV01_radial.nc
did not get uploaded correctly when they were recently reprocessed due to an error on our end and thus were bad files with a much older time stamp compared to adjacent files in AODN, so perhaps this caused the problem? I have uploaded the new corrected versions today. So hopefully this resolves the issue.

@scosoli, could you check the DONG file? I think you created that radial file.

[ocehugo]

Thanks guys, the only missing now is the DONG one.

@ocehugo, I don't see how IMOS_ACORN_V_20090327T090000Z_TURQ_FV00_1-hour-avg.nc has global attributes different from or missing in comparison to other files. Say, compare for instance with IMOS_ACORN_V_20090327T100000Z_TURQ_FV00_1-hour-avg.nc or IMOS_ACORN_V_20090327T110000Z_TURQ_FV00_1-hour-avg.nc.
can you provide more details?

sorry, I should have been more specific - this one was just for a double-check since the published file was OK and slightly newer than the errored one.

I don't see a problem with IMOS_ACORN_RV_20180105T144000Z_CWI_FV01_radial.nc either and have re-uploaded the file I have locally.

As above - just a double-check. I think this one was a fail on our side.

[ocehugo]

Update the list in here: #474