Unclear why Files SKIPPED

[sashakames]

For some files, specifically from the MCM UA model, we see this external_variables message followed by the file being skipped. Is there more information that can be provided here?

mon_MCM-UA-1-0_historical_r1i1p1f1_gn_185001-201412.nc...
ESC[1;32m^MCheck netCDF file(s): ESC[0m100% | 1/1 files^MESC[K'external_variables'
ESC[1;34;47m└──> :: SKIPPED    :: /p/css03/sandbox/ingest/historical/r1i1p1f1/Amon/tas/gn/v20190731/tas_Amon_MCM-UA-1-0_historical_r1i1p1f1_gn_185001-201412.nc
ESC[0m
ESC[95m
Number of files scanned: 1ESC[0mESC[1;31;47m
Number of file with error(s): 1ESC[0m

[taylor13]

@mauzey1 has confirmed that this means that the global attribute external_variables should be defined in this file, so could the error message please say something more helpful to the user like

PrePARE found error in /p/css03/sandbox/ingest/historical/r1i1p1f1/Amon/tas/gn/v20190731/tas_Amon_MCM-UA-1-0_historical_r1i1p1f1_gn_185001-201412.nc
This file is missing the following required global attribute: 'external_variables'.  
This file cannot be published.

[taylor13]

@sashakames @mauzey1 I think the error message should be replaced with the above. Does that need to be done in CMOR/PrePARE, or in ESGF publisher?

[sashakames]

This has to be a PrePARE generated message. The publisher will just detect a success or failure, and in the case of failure halts. So important to define what is unacceptable for publication and throw an error in that case.

[sashakames]

To follow up (that this really aught to fixed) we encounter on an additional file when PrePARE is integrated with esg-publisher:

File "/export/ames4/.conda/envs/pub-py3/lib/python3.7/site-packages/cmip6_cv/P
rePARE/PrePARE.py", line 604, in ControlVocab
if not re.search(r"\b{}\b".format(v), self.dictGbl['external_variables']):
KeyError: 'external_variables'

example file: CMIP6/HighResMIP/MPI-M/MPI-ESM1-2-XR/highresSST-present/r1i1p1f1/6hrPlev/psl/gn/v20171003/psl_6hrPlev_MPI-ESM1-2-XR_highresSST-present_r1i1p1f1_gn_195001010558-195012312358.nc

[mauzey1]

There is an issue with PrePARE when files have area and volume values in cell_measures but do not have external_variables in their global attributes. This will cause an error when trying to check if the area and volume values are in external_variables.

cmor/LibCV/PrePARE/PrePARE.py

Lines 599 to 615 in 7c252cc

	if key == "cell_measures":
	# Check if area and volume values from the table's cell_measures are found in the file's external_variables
	pattern = re.compile('(?:area|volume): (\w+)')
	values = re.findall(pattern, table_value)
	for v in values:
	if not re.search(r"\b{}\b".format(v), self.dictGbl['external_variables']):
	print(BCOLORS.FAIL)
	print("=====================================================================================")
	print("Your file contains external_variables = \"" + self.dictGbl['external_variables'] + "\", and")
	if len(values) == 2:
	print("CMIP6 tables requires \"" + values[0] + "\" and \"" + values[1] + "\" in external_variables.")
	else:
	print("CMIP6 tables requires \"" + values[0] + "\" in external_variables.")
	print("=====================================================================================")
	print(BCOLORS.ENDC)
	self.errors += 1
	continue

I have added another test to see if external_variables exists before accessing it. Here is an example of the output.

$ PrePARE ~/Downloads/psl_6hrPlev_MPI-ESM1-2-XR_highresSST-present_r1i1p1f1_gn_195001010558-195012312358.nc 

=====================================================================================
Your file contains "standard_name":"air_pressure_at_sea_level" and
CMIP6 tables requires "standard_name":"air_pressure_at_mean_sea_level".
=====================================================================================


=====================================================================================
Your file contains "areacella" in cell_measures and
CMIP6 tables require attribute "external_variables" in global attributes.
=====================================================================================

└──> :: CV FAIL    :: /home/mauzey1/Downloads/psl_6hrPlev_MPI-ESM1-2-XR_highresSST-present_r1i1p1f1_gn_195001010558-195012312358.nc

Number of files scanned: 1
Number of file with error(s): 1