moda_oc_7d_gom failing

[7yl4r]

Dataset is failing according to status.html.

DasDds output:

[tylar@dune ~]$ docker exec -it erddap bash -c "cd webapps/erddap/WEB-INF/ && bash DasDds.sh -verbose"

////**** EDStatic Low Level Startup
localTime=2023-03-09T16:54:39+00:00
erddapVersion=2.18
Java 1.8.0_322 (64 bit, Oracle Corporation) on Linux (4.18.0-408.el8.x86_64).
MemoryInUse=    50 MB (highWaterMark=    50 MB) (Xmx ~= 958 MB)
logLevel=info: verbose=true reallyVerbose=false
got bigParentDirectory from ERDDAP_bigParentDirectory
got emailFromAddress from ERDDAP_emailFromAddress
got emailEverythingTo from ERDDAP_emailEverythingTo
ERROR in File2.deleteIfOld: dir=/erddapData/dataset/_FileVisitor/ isn't a directory.
bigParentDirectory=/erddapData/
webInfParentDirectory=/usr/local/tomcat/webapps/erddap/
java.awt.HeadlessException: 
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
 at gov.noaa.pfel.coastwatch.sgt.SgtUtil.isBufferedImageAccelerated(SgtUtil.java:368)
 at gov.noaa.pfel.erddap.util.EDStatic.<clinit>(EDStatic.java:1830)
 at gov.noaa.pfel.erddap.DasDds.<clinit>(DasDds.java:31)


bufferedImage isAccelerated=[unknown]
got baseHttpsUrl from ERDDAP_baseHttpsUrl
got adminInstitution from ERDDAP_adminInstitution
got adminInstitutionUrl from ERDDAP_adminInstitutionUrl
got adminIndividualName from ERDDAP_adminIndividualName
got adminPosition from ERDDAP_adminPosition
got adminPhone from ERDDAP_adminPhone
got adminAddress from ERDDAP_adminAddress
got adminCity from ERDDAP_adminCity
got adminStateOrProvince from ERDDAP_adminStateOrProvince
got adminPostalCode from ERDDAP_adminPostalCode
got adminCountry from ERDDAP_adminCountry
got adminEmail from ERDDAP_adminEmail
got flagKeyKey from ERDDAP_flagKeyKey
  copying images/ file: erddapStart2.css
Custom messages.xml not found at /usr/local/tomcat/content/erddap/messages.xml
Using default messages.xml from  /usr/local/tomcat/webapps/erddap/WEB-INF/classes/gov/noaa/pfel/erddap/util/messages.xml
CfToFromGcmd static loading /usr/local/tomcat/webapps/erddap/WEB-INF/classes/gov/noaa/pfel/erddap/util/CfToGcmd.txt
*** EDStatic Low Level Startup finished successfully.

logFileMaxSize=20000000
*** Starting DasDds 2023-03-09T16:54:41+00:00 erddapVersion=2.18
logFile=/erddapData/logs/DasDds.log
Java 1.8.0_322 (64 bit, Oracle Corporation) on Linux (4.18.0-408.el8.x86_64).
MemoryInUse=   170 MB (highWaterMark=   170 MB) (Xmx ~= 958 MB)
verbose=true

*** DasDds ***
This generates the DAS and DDS for a dataset and puts it in
/erddapData/logs/DasDds.out
Press ^D or ^C to exit at any time.

Which datasetID (default="")
? moda_oc_7d_gom

*** DasDds moda_oc_7d_gom
*** deleting cached dataset info for datasetID=moda_oc_7d_gom
File2.deleteIfOld(/erddapData/dataset/om/moda_oc_7d_gom/                 ) nDir=   0 nDeleted=   4 nRemain=   0

EDD.oneFromDatasetsXml(moda_oc_7d_gom)...
SimpleXmlReader encoding=UTF-8

*** constructing EDDGridFromFiles(xmlReader)...

*** constructing EDDGridFromFiles moda_oc_7d_gom
axis0 ***fileName format=yyyyDDD class=DOUBLE regex=MODA_(\d{7})_(\d{7})_7D_GOM_OC\.nc captureGroup=2
sourceAxisNames="***fileName,timeFormat=yyyyDDD,MODA_(\\d{7})_(\\d{7})_7D_GOM_OC\\.nc,2", latitude, longitude
sourceDataNames=chlor_a_median, chlor_a_anom, ABI_median, ABI_anom, Rrs_667_median, Rrs_667_anom
sourceDataTypes=double, double, double, double, double, double
dir/file table doesn't exist: /erddapData/dataset/om/moda_oc_7d_gom/dirTable.nc
dir/file table doesn't exist: /erddapData/dataset/om/moda_oc_7d_gom/fileTable.nc
creating new dirTable and fileTable (dirTable=null?true fileTable=null?true badFileMap=null?false)
doQuickRestart=false
1041 files found in /mnt/sdb/moda_oc_7d_gom/
regex=.*\.nc recursive=true pathRegex=.* time=41ms
old nBadFiles size=0
sortTime=3ms
old fileTable size=0   nFilesMissing=0
#0 file=/mnt/sdb/moda_oc_7d_gom/MODA_2002185_2002189_7D_GOM_OC.nc
#0 inserted in cache
#1 file=/mnt/sdb/moda_oc_7d_gom/MODA_2002190_2002196_7D_GOM_OC.nc
#1 inserted in cache
#2 file=/mnt/sdb/moda_oc_7d_gom/MODA_2002197_2002203_7D_GOM_OC.nc
#2 inserted in cache
[...]
#1039 file=/mnt/sdb/moda_oc_7d_gom/MODA_2022197_2022203_7D_GOM_OC.nc
#1039 inserted in cache
#1040 file=/mnt/sdb/moda_oc_7d_gom/MODA_2022204_2022210_7D_GOM_OC.nc
#1040 inserted in cache
fileTable updated; time=1912ms
2nd sortTime=2ms

  tFileNamePA.size=1041
  dirTable.nRows=1
  fileTable.nRows=1041
    fileTableInMemory=false
    nUnchanged=0
    nRemoved=0 (nNoLastMod=0, nNoSize=0)
    nReadFile=1041 (nDifferentModTime=0 nNew=1041) readFileCumTime=1.059 s avg=1ms
  Table.saveAsFlatNc /erddapData/dataset/om/moda_oc_7d_gom/dirTable.nc457116491 done. nColumns=1 nRows=1 TIME=16ms
  Table.saveAsFlatNc /erddapData/dataset/om/moda_oc_7d_gom/fileTable.nc457116491 done. nColumns=9 nRows=1041 TIME=34ms
save fileTable(first 5 rows)=
dirIndex,fileList,lastMod,size,nValues,min,max,csvValues,startIndex
0,MODA_2002185_2002189_7D_GOM_OC.nc,1675979484476,147262002,1,1.0260864E9,1.0260864E9,1.0260864E9,0
0,MODA_2002190_2002196_7D_GOM_OC.nc,1675979485748,147262002,1,1.0266912E9,1.0266912E9,1.0266912E9,1
0,MODA_2002197_2002203_7D_GOM_OC.nc,1675979487033,147262002,1,1.027296E9,1.027296E9,1.027296E9,2
0,MODA_2002204_2002210_7D_GOM_OC.nc,1675979488364,147262002,1,1.0279008E9,1.0279008E9,1.0279008E9,3
0,MODA_2002219_2002224_7D_GOM_OC.nc,1675979489773,147262002,1,1.0291104E9,1.0291104E9,1.0291104E9,4
...

getting metadataFrom /mnt/sdb/moda_oc_7d_gom/MODA_2022204_2022210_7D_GOM_OC.nc
  ftLastMod first=2023-02-09T21:51:24Z last=2023-02-09T22:15:13Z
  time: DoubleArray isn't evenly spaced: [0]=1.0260864E9, [1]=1.0266912E9, spacing=604800.0, average spacing=608621.5384615385.
    smallest spacing=432000.0: [1024]=1.6481664E9, [1025]=1.6485984E9
    biggest  spacing=1987200.0: [1025]=1.6485984E9, [1026]=1.6505856E9
  latitude: FloatArray isn't sorted in ascending order: [0]=31.0 > [1]=30.990911.

*** EDDGridFromFiles moda_oc_7d_gom constructor finished. TIME=2101ms

  EDDGrid.saveAsDAS
  EDDGrid.saveAsDAS done. TIME=4ms

  EDDGrid.writeDDS
    EDDGrid.parseDataDapQuery: 
      query="" is expanded to request entire dataset.
  Table.readFlatNc /erddapData/dataset/om/moda_oc_7d_gom/dirTable.nc finished. nColumns=1 nRows=1 TIME=15ms
  Table.readFlatNc /erddapData/dataset/om/moda_oc_7d_gom/fileTable.nc finished. nColumns=9 nRows=1041 TIME=18ms
  EDDGrid.writeDDS done. TIME=53ms

**************************** The .das for moda_oc_7d_gom ****************************
Attributes {
  time {
    String _CoordinateAxisType "Time";
    Float64 actual_range 1.0260864e+9, 1.6590528e+9;
    String axis "T";
    String ioos_category "Time";
    String long_name "Time";
    String standard_name "time";
    String time_origin "01-JAN-1970 00:00:00";
    String units "seconds since 1970-01-01T00:00:00Z";
  }
  latitude {
    String _CoordinateAxisType "Lat";
    Float32 actual_range 18.01288, 31.0;
    String axis "Y";
    String ioos_category "Location";
    String long_name "Latitude";
    String standard_name "latitude";
    String units "degrees_north";
  }
  longitude {
    String _CoordinateAxisType "Lon";
    Float32 actual_range -98.0, -78.51478;
    String axis "X";
    String ioos_category "Location";
    String long_name "Longitude";
    String standard_name "longitude";
    String units "degrees_east";
  }
  chlor_a_median {
    Float64 colorBarMaximum 20.0;
    Float64 colorBarMinimum 0.01;
    String colorBarScale "Log";
    String ioos_category "Ocean Color";
    String long_name "Concentration Of Chlorophyll In Sea Water";
    String Product "chlor_a";
    String standard_name "concentration_of_chlorophyll_in_sea_water";
    String units "mg m^-3";
  }
  chlor_a_anom {
    Float64 colorBarMaximum 1.0;
    Float64 colorBarMinimum -1.0;
    String ioos_category "Ocean Color";
    String long_name "Chlor A Anom";
    String Product "chlor_a_anomaly";
    String units "mg m^-3";
  }
  ABI_median {
    Float64 colorBarMaximum 0.5;
    Float64 colorBarMinimum 0.0;
    String ioos_category "Unknown";
    String long_name "ABI Median";
    String Product "ABI";
    String units "W m^-2 um^-1 sr^-1";
  }
  ABI_anom {
    Float64 colorBarMaximum 0.2;
    Float64 colorBarMinimum -0.2;
    String ioos_category "Unknown";
    String long_name "ABI Anom";
    String Product "ABI_anomaly";
    String units "W m^-2 um^-1 sr^-1";
  }
  Rrs_667_median {
    Float64 colorBarMaximum 0.01;
    Float64 colorBarMinimum 0.0;
    String ioos_category "Optical Properties";
    String long_name "Rrs 667 Median";
    String Product "Rrs_667";
    String units "sr^-1";
  }
  Rrs_667_anom {
    Float64 colorBarMaximum 0.001;
    Float64 colorBarMinimum -0.001;
    String ioos_category "Optical Properties";
    String long_name "Rrs 667 Anom";
    String Product "Rrs_667_anomaly";
    String units "sr^-1";
  }
  NC_GLOBAL {
    String cdm_data_type "Grid";
    String Composite_end_date "07/29/2022";
    String Composite_start_date "07/23/2022";
    String contact "Dan Otis - dotis@usf.edu";
    String Conventions "COARDS, CF-1.6, ACDD-1.3";
    String CreationDate "09/09/2022 08:39:22";
    String creator_email "dotis@usf.edu";
    String creator_name "DOTIS";
    String creator_type "institution";
    Float64 Easternmost_Easting -78.51478;
    Float64 geospatial_lat_max 31.0;
    Float64 geospatial_lat_min 18.01288;
    Float64 geospatial_lat_resolution 0.009088257522743177;
    String geospatial_lat_units "degrees_north";
    Float64 geospatial_lon_max -78.51478;
    Float64 geospatial_lon_min -98.0;
    Float64 geospatial_lon_resolution 0.009088255597014925;
    String geospatial_lon_units "degrees_east";
    String history "2023-03-09T16:55:21Z (local files)
2023-03-09T16:55:21Z http://131.247.136.200:8080/erddap/griddap/moda_oc_7d_gom.das";
    String Image_size "1430 pixels(N-S) x 2145 pixels(E-W)";
    String infoUrl "imars.usf.edu";
    String institution "USF IMaRS";
    String keywords "abi, ABI_anom, ABI_median, anomaly, chemistry, chlor, chlor_a_anom, chlor_a_median, chlorophyll, color, concentration, concentration_of_chlorophyll_in_sea_water, data, earth, Earth Science > Oceans > Ocean Chemistry > Chlorophyll, florida, img, img_x, img_y, local, median, ocean, ocean color, oceans, optical, optical properties, properties, rrs, Rrs_667_anom, Rrs_667_median, science, sea, seawater, source, south, university, usf, water";
    String keywords_vocabulary "GCMD Science Keywords";
    String Lat_Lon_Limits "18.0129N to 31N -98W to -78.5148W";
    String license "The data may be used and redistributed for free but is not intended
for legal use, since it may contain inaccuracies. Neither the data
Contributor, ERD, NOAA, nor the United States Government, nor any
of their employees or contractors, makes any warranty, express or
implied, including warranties of merchantability and fitness for a
particular purpose, or assumes any legal liability for the accuracy,
completeness, or usefulness, of this information.";
    Float64 Northernmost_Northing 31.0;
    String Ocean_color_masks_based_on_L2_flags "LAND,CLDICE,HIGLINT";
    String Original_Image_Format "Level-2(NetCDF)";
    String Original_Image_Source "NASA Ocean Biology Processing Group";
    String Processing_and_binning "USF IMaRS";
    String Projection "Equidistant Cylindrical";
    String Region "Gulf of Mexico (GOM)";
    String Sensor "MODIS-Aqua";
    String sourceUrl "(local files)";
    Float64 Southernmost_Northing 18.01288;
    String standard_name_vocabulary "CF Standard Name Table v70";
    String summary "USF IMaRS MODIS FK 7 Day Mean Ocean Color.";
    String time_coverage_end "2022-07-29T00:00:00Z";
    String time_coverage_start "2002-07-08T00:00:00Z";
    String Time_interval "7-Day Composite (median)";
    String title "IMaRS MODA 7D FK..";
    Float64 Westernmost_Easting -98.0;
  }
}

**************************** The .dds for moda_oc_7d_gom ****************************
Dataset {
  Float64 time[time = 1041];
  Float32 latitude[latitude = 1430];
  Float32 longitude[longitude = 2145];
  GRID {
    ARRAY:
      Float64 chlor_a_median[time = 1041][latitude = 1430][longitude = 2145];
    MAPS:
      Float64 time[time = 1041];
      Float32 latitude[latitude = 1430];
      Float32 longitude[longitude = 2145];
  } chlor_a_median;
  GRID {
    ARRAY:
      Float64 chlor_a_anom[time = 1041][latitude = 1430][longitude = 2145];
    MAPS:
      Float64 time[time = 1041];
      Float32 latitude[latitude = 1430];
      Float32 longitude[longitude = 2145];
  } chlor_a_anom;
  GRID {
    ARRAY:
      Float64 ABI_median[time = 1041][latitude = 1430][longitude = 2145];
    MAPS:
      Float64 time[time = 1041];
      Float32 latitude[latitude = 1430];
      Float32 longitude[longitude = 2145];
  } ABI_median;
  GRID {
    ARRAY:
      Float64 ABI_anom[time = 1041][latitude = 1430][longitude = 2145];
    MAPS:
      Float64 time[time = 1041];
      Float32 latitude[latitude = 1430];
      Float32 longitude[longitude = 2145];
  } ABI_anom;
  GRID {
    ARRAY:
      Float64 Rrs_667_median[time = 1041][latitude = 1430][longitude = 2145];
    MAPS:
      Float64 time[time = 1041];
      Float32 latitude[latitude = 1430];
      Float32 longitude[longitude = 2145];
  } Rrs_667_median;
  GRID {
    ARRAY:
      Float64 Rrs_667_anom[time = 1041][latitude = 1430][longitude = 2145];
    MAPS:
      Float64 time[time = 1041];
      Float32 latitude[latitude = 1430];
      Float32 longitude[longitude = 2145];
  } Rrs_667_anom;
} moda_oc_7d_gom;

************************* The .timeGaps for moda_oc_7d_gom *************************
Time gaps greater than the median (7 days):
[3]=2002-07-29T00:00:00Z -> [4]=2002-08-12T00:00:00Z, gap=14 days
[8]=2002-09-08T00:00:00Z -> [9]=2002-09-16T00:00:00Z, gap=8 days
[23]=2002-12-23T00:00:00Z -> [24]=2002-12-31T00:00:00Z, gap=8 days
[75]=2003-12-23T00:00:00Z -> [76]=2003-12-31T00:00:00Z, gap=8 days
[127]=2004-12-22T00:00:00Z -> [128]=2004-12-31T00:00:00Z, gap=9 days
[179]=2005-12-23T00:00:00Z -> [180]=2005-12-31T00:00:00Z, gap=8 days
[231]=2006-12-23T00:00:00Z -> [232]=2006-12-31T00:00:00Z, gap=8 days
[280]=2007-12-01T00:00:00Z -> [281]=2007-12-09T00:00:00Z, gap=8 days
[283]=2007-12-23T00:00:00Z -> [284]=2007-12-31T00:00:00Z, gap=8 days
[335]=2008-12-22T00:00:00Z -> [336]=2008-12-31T00:00:00Z, gap=9 days
[387]=2009-12-23T00:00:00Z -> [388]=2009-12-31T00:00:00Z, gap=8 days
[439]=2010-12-23T00:00:00Z -> [440]=2010-12-31T00:00:00Z, gap=8 days
[491]=2011-12-23T00:00:00Z -> [492]=2011-12-31T00:00:00Z, gap=8 days
[543]=2012-12-22T00:00:00Z -> [544]=2012-12-31T00:00:00Z, gap=9 days
[595]=2013-12-23T00:00:00Z -> [596]=2013-12-31T00:00:00Z, gap=8 days
[647]=2014-12-23T00:00:00Z -> [648]=2014-12-31T00:00:00Z, gap=8 days
[699]=2015-12-23T00:00:00Z -> [700]=2015-12-31T00:00:00Z, gap=8 days
[751]=2016-12-22T00:00:00Z -> [752]=2016-12-31T00:00:00Z, gap=9 days
[803]=2017-12-23T00:00:00Z -> [804]=2017-12-31T00:00:00Z, gap=8 days
[855]=2018-12-23T00:00:00Z -> [856]=2018-12-31T00:00:00Z, gap=8 days
[907]=2019-12-23T00:00:00Z -> [908]=2019-12-31T00:00:00Z, gap=8 days
[959]=2020-12-22T00:00:00Z -> [960]=2020-12-31T00:00:00Z, gap=9 days
[1011]=2021-12-23T00:00:00Z -> [1012]=2021-12-31T00:00:00Z, gap=8 days
[1025]=2022-03-30T00:00:00Z -> [1026]=2022-04-22T00:00:00Z, gap=23 days
nGaps=24

---

Possible issues from the DasDds output:

java.awt.HeadlessException

This is expected and not related

time variable spacing

DasDds is complaining about the spacing of the time variable being uneven, but this is common and should not be breaking the dataset.

  time: DoubleArray isn't evenly spaced: [0]=1.0260864E9, [1]=1.0266912E9, spacing=604800.0, average spacing=608621.5384615385.
    smallest spacing=432000.0: [1024]=1.6481664E9, [1025]=1.6485984E9
    biggest  spacing=1987200.0: [1025]=1.6485984E9, [1026]=1.6505856E9

latitude needs to be in ascending order

  latitude: FloatArray isn't sorted in ascending order: [0]=31.0 > [1]=30.990911.

I think this is the issue. The latitude variable needs to be flipped around.

@dotis: Can we try that and see if the updated files work with ERDDAP?

[7yl4r]

Dan has reprocessed the files and flipped the latitude. The dataset is still showing as failed in the status.html, but I don't see any big errors in the DasDds output.

Based on this maybe it has something to do with permissions? The files are publicly readable on the hypervisor:

[tylar@dune ~]$ ls -lah /mnt/sdb/moda_oc_7d_gom/MODA_2005134_2005140_7D_GOM_OC.nc
-rw-r--r--. 1 dotis imars-common 141M Mar 10 17:17 /mnt/sdb/moda_oc_7d_gom/MODA_2005134_2005140_7D_GOM_OC.nc

---

I just followed these instructions to resolve possible SELINUX issues.

The files appear readable in the container:

[tylar@dune ~]$ docker exec -it erddap bash -c "ls -lah /mnt/sdb/moda_oc_7d_gom/MODA_2005134_2005140_7D_GOM_OC.nc"
-rw-r--r--. 1 4509 4504 141M Mar 10 22:17 /mnt/sdb/moda_oc_7d_gom/MODA_2005134_2005140_7D_GOM_OC.nc

[7yl4r]

I don't see anything helpful from the erddap logs in docker:

[tylar@dune ~]$ docker logs erddap 2>&1 | grep -Ev "IllegalArgument|Http11Processor|at org.|at java.|Note:"
ERDDAP configured with:
ERDDAP_MEMORY=64G
ERDDAP_adminAddress=123 Irrelevant St.
ERDDAP_adminCity=Nowhere
ERDDAP_adminCountry=USA
ERDDAP_adminEmail=nobody@example.com
ERDDAP_adminIndividualName=Axiom Docker Install
ERDDAP_adminInstitution=Axiom Docker Install
ERDDAP_adminInstitutionUrl=https://github.com/axiom-data-science/docker-erddap
ERDDAP_adminPhone=555-555-5555
ERDDAP_adminPosition=Software Engineer
ERDDAP_adminPostalCode=99504
ERDDAP_adminStateOrProvince=AK
ERDDAP_baseHttpsUrl=https://localhost:8443
ERDDAP_bigParentDirectory=/erddapData
ERDDAP_emailDailyReportsTo=nobody@example.com
ERDDAP_emailEverythingTo=nobody@example.com
ERDDAP_emailFromAddress=nothing@example.com
ERDDAP_emailPassword=
ERDDAP_emailProperties=
ERDDAP_emailSmtpHost=
ERDDAP_emailSmtpPort=
ERDDAP_emailUserName=
ERDDAP_flagKeyKey=73976bb0-9cd4-11e3-a5e2-0800200c9a66
ERDDAP Running with:  -server -Xms64G -Xmx64G -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true  -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -DerddapContentDirectory=/usr/local/tomcat/content/erddap/ -Djna.tmpdir=/tmp/ -Djava.security.egd=file:/dev/./urandom
13-Jan-2023 16:41:07.070 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name:   Apache Tomcat
13-Jan-2023 16:41:07.073 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Jan 17 2022 22:07:47 UTC
13-Jan-2023 16:41:07.073 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 8.5.75.0
13-Jan-2023 16:41:07.073 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Linux
13-Jan-2023 16:41:07.073 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            4.18.0-408.el8.x86_64
13-Jan-2023 16:41:07.073 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
13-Jan-2023 16:41:07.074 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /usr/local/openjdk-8/jre
13-Jan-2023 16:41:07.074 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           1.8.0_322-b06
13-Jan-2023 16:41:07.074 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Oracle Corporation
13-Jan-2023 16:41:07.074 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /usr/local/tomcat
13-Jan-2023 16:41:07.074 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /usr/local/tomcat
13-Jan-2023 16:41:07.075 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties
13-Jan-2023 16:41:07.075 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
13-Jan-2023 16:41:07.075 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms64G
13-Jan-2023 16:41:07.075 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx64G
13-Jan-2023 16:41:07.075 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+HeapDumpOnOutOfMemoryError
13-Jan-2023 16:41:07.075 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.awt.headless=true
13-Jan-2023 16:41:07.079 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UseConcMarkSweepGC
13-Jan-2023 16:41:07.079 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+CMSClassUnloadingEnabled
13-Jan-2023 16:41:07.079 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -DerddapContentDirectory=/usr/local/tomcat/content/erddap/
13-Jan-2023 16:41:07.079 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djna.tmpdir=/tmp/
13-Jan-2023 16:41:07.079 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.security.egd=file:/dev/./urandom
13-Jan-2023 16:41:07.079 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
13-Jan-2023 16:41:07.080 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
13-Jan-2023 16:41:07.080 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
13-Jan-2023 16:41:07.080 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
13-Jan-2023 16:41:07.080 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/usr/local/tomcat
13-Jan-2023 16:41:07.080 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/usr/local/tomcat
13-Jan-2023 16:41:07.080 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/usr/local/tomcat/temp
13-Jan-2023 16:41:07.081 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded Apache Tomcat Native library [1.2.31] using APR version [1.7.0].
13-Jan-2023 16:41:07.081 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true], UDS [{4}].
13-Jan-2023 16:41:07.081 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
13-Jan-2023 16:41:07.084 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.1.1k  25 Mar 2021]
13-Jan-2023 16:41:07.161 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
13-Jan-2023 16:41:07.175 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
13-Jan-2023 16:41:07.192 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 699 ms
13-Jan-2023 16:41:07.225 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
13-Jan-2023 16:41:07.226 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat]
13-Jan-2023 16:41:07.234 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/usr/local/tomcat/webapps/erddap]
13-Jan-2023 16:41:14.199 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
13-Jan-2023 16:41:14.248 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/erddap] has finished in [7,013] ms
13-Jan-2023 16:41:14.251 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
13-Jan-2023 16:41:14.261 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 7069 ms

////**** EDStatic Low Level Startup
localTime=2023-01-13T16:41:37+00:00
erddapVersion=2.18
Java 1.8.0_322 (64 bit, Oracle Corporation) on Linux (4.18.0-408.el8.x86_64).
MemoryInUse=   845 MB (highWaterMark=   845 MB) (Xmx ~= 65303 MB)
logLevel=info: verbose=true reallyVerbose=false
got bigParentDirectory from ERDDAP_bigParentDirectory
got emailFromAddress from ERDDAP_emailFromAddress
got emailEverythingTo from ERDDAP_emailEverythingTo
ERROR in File2.deleteIfOld: dir=/erddapData/dataset/_FileVisitor/ isn't a directory.
bigParentDirectory=/erddapData/
webInfParentDirectory=/usr/local/tomcat/webapps/erddap/
java.awt.HeadlessException
 at gov.noaa.pfel.coastwatch.sgt.SgtUtil.isBufferedImageAccelerated(SgtUtil.java:368)
 at gov.noaa.pfel.erddap.util.EDStatic.<clinit>(EDStatic.java:1830)
 at gov.noaa.pfel.erddap.Erddap.<init>(Erddap.java:223)


bufferedImage isAccelerated=[unknown]
got baseHttpsUrl from ERDDAP_baseHttpsUrl
got adminInstitution from ERDDAP_adminInstitution
got adminInstitutionUrl from ERDDAP_adminInstitutionUrl
got adminIndividualName from ERDDAP_adminIndividualName
got adminPosition from ERDDAP_adminPosition
got adminPhone from ERDDAP_adminPhone
got adminAddress from ERDDAP_adminAddress
got adminCity from ERDDAP_adminCity
got adminStateOrProvince from ERDDAP_adminStateOrProvince
got adminPostalCode from ERDDAP_adminPostalCode
got adminCountry from ERDDAP_adminCountry
got adminEmail from ERDDAP_adminEmail
got flagKeyKey from ERDDAP_flagKeyKey
  copying images/ file: erddapStart2.css
Custom messages.xml not found at /usr/local/tomcat/content/erddap/messages.xml
Using default messages.xml from  /usr/local/tomcat/webapps/erddap/WEB-INF/classes/gov/noaa/pfel/erddap/util/messages.xml
CfToFromGcmd static loading /usr/local/tomcat/webapps/erddap/WEB-INF/classes/gov/noaa/pfel/erddap/util/CfToGcmd.txt
*** EDStatic Low Level Startup finished successfully.

06-Mar-2023 12:07:22.181 INFO [http-nio-8080-exec-16] org.apache.coyote.AbstractProcessor.parseHost The host [www.$$$$$] is not valid

I don't know the importance of ERROR in File2.deleteIfOld: dir=/erddapData/dataset/_FileVisitor/ isn't a directory. but this looks to be a global issue, not one specific to this dataset.

[7yl4r]

This issue seems to be similar to this unresolved issue with an older dataset.

Note that NFS is no longer a factor here, because these files are on a local drive (/mnt/sdb).

[dotis]

Hmm. I don't think I can help much with this.

…

________________________________ From: Tylar ***@***.***> Sent: Tuesday, March 14, 2023 11:32 AM To: USF-IMARS/erddap-config ***@***.***> Cc: dotis ***@***.***>; Mention ***@***.***> Subject: Re: [USF-IMARS/erddap-config] moda_oc_7d_gom failing (Issue #35) This issue seems to be similar to this unresolved issue with an older dataset<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmarinebon%2Fmbon-dashboard-server%2Fissues%2F36&data=05%7C01%7Cdotis%40usf.edu%7C15c5a5e02fe445351ba808db24a15741%7C741bf7dee2e546df8d6782607df9deaa%7C0%7C0%7C638144047593163430%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=eV94X8xzNq9rWxA7BDLjsczibfSPTGfFxm%2B9BRQjW%2Bg%3D&reserved=0>. Note that NFS is no longer a factor here, because these files are on a local drive (/mnt/sdb). — Reply to this email directly, view it on GitHub<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FUSF-IMARS%2Ferddap-config%2Fissues%2F35%23issuecomment-1468326570&data=05%7C01%7Cdotis%40usf.edu%7C15c5a5e02fe445351ba808db24a15741%7C741bf7dee2e546df8d6782607df9deaa%7C0%7C0%7C638144047593163430%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=rFBZIE3RnTHRLLjeIGsNq96a84mw5mXkL4s%2BZO0Dc2w%3D&reserved=0>, or unsubscribe<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAGKS3DNQQPVO5B42YLYUV5TW4CFRJANCNFSM6AAAAAAVVMEIGA&data=05%7C01%7Cdotis%40usf.edu%7C15c5a5e02fe445351ba808db24a15741%7C741bf7dee2e546df8d6782607df9deaa%7C0%7C0%7C638144047593163430%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=c4lhfyY8v0RRH4RG7fAu2m6GscFHyFc2dnuM8jXtJWQ%3D&reserved=0>. You are receiving this because you were mentioned.Message ID: ***@***.***> [EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe.

[7yl4r]

found the error in /erddapData/logs/log.txt:

Emailing "86861bdaf3af ERDDAP: ERROR while saving dirTable, fileTable, or badFiles for moda_oc_7d_gom" to 
datasets.xml error on line #717
While trying to load datasetID=moda_oc_7d_gom (after 17 ms)
java.lang.RuntimeException: datasets.xml error on or before line #717: FileNotFoundException: /erddapData/dataset/om/moda_oc_7d_gom/dirTable.nc93412081725963838 (Permission denied)
 at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:471)
 at gov.noaa.pfel.erddap.LoadDatasets.run(LoadDatasets.java:359)
Caused by: java.io.FileNotFoundException: /erddapData/dataset/om/moda_oc_7d_gom/dirTable.nc93412081725963838 (Permission denied)
 at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:322)
 at ucar.unidata.io.RandomAccessFile.<init>(RandomAccessFile.java:301)
 at ucar.nc2.internal.iosp.netcdf3.N3iospWriter.create(N3iospWriter.java:84)
 at ucar.nc2.write.NetcdfFormatWriter.<init>(NetcdfFormatWriter.java:321)
 at ucar.nc2.write.NetcdfFormatWriter.<init>(NetcdfFormatWriter.java:42)
 at ucar.nc2.write.NetcdfFormatWriter$Builder.build(NetcdfFormatWriter.java:251)
 at gov.noaa.pfel.coastwatch.pointdata.Table.saveAsFlatNc(Table.java:28685)
 at gov.noaa.pfel.coastwatch.pointdata.Table.saveAsEnhancedFlatNc(Table.java:6081)
 at gov.noaa.pfel.erddap.dataset.EDD.saveDirTableFileTableBadFiles(EDD.java:10603)
 at gov.noaa.pfel.erddap.dataset.EDDGridFromFiles.<init>(EDDGridFromFiles.java:1125)
 at gov.noaa.pfel.erddap.dataset.EDDGridFromNcLow.<init>(EDDGridFromNcLow.java:96)
 at gov.noaa.pfel.erddap.dataset.EDDGridFromNcFiles.<init>(EDDGridFromNcFiles.java:102)
 at gov.noaa.pfel.erddap.dataset.EDDGridFromFiles.fromXml(EDDGridFromFiles.java:299)
 at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:430)
 ... 1 more

This could be a larger issue with the setup (or lack of setup) of the /erddapData directory in the docker-compose.yml file. I am going to attempt a fix, docker-compose down --rmi all --volumes && docker-compose up -d, then check back in at least an hour.

[7yl4r]

Yep, that fixed it.

http://131.247.136.200:8080/erddap/griddap/moda_oc_7d_gom.graph

[dotis]

Upside down! I knew it. I will re-orient. I take it the lat was not the issue.

…

________________________________ From: Tylar ***@***.***> Sent: Tuesday, March 14, 2023 1:22 PM To: USF-IMARS/erddap-config ***@***.***> Cc: dotis ***@***.***>; Mention ***@***.***> Subject: Re: [USF-IMARS/erddap-config] moda_oc_7d_gom failing (Issue #35) Yep, that fixed it. http://131.247.136.200:8080/erddap/griddap/moda_oc_7d_gom.graph<https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2F131.247.136.200%3A8080%2Ferddap%2Fgriddap%2Fmoda_oc_7d_gom.graph&data=05%7C01%7Cdotis%40usf.edu%7C197854ee262f4e7d90ae08db24b0c1db%7C741bf7dee2e546df8d6782607df9deaa%7C0%7C0%7C638144113824006482%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=uGLknt0QqnRCe8ioLKT5SVne%2BAoFE%2BJeRffyP7yK9v0%3D&reserved=0> — Reply to this email directly, view it on GitHub<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FUSF-IMARS%2Ferddap-config%2Fissues%2F35%23issuecomment-1468518738&data=05%7C01%7Cdotis%40usf.edu%7C197854ee262f4e7d90ae08db24b0c1db%7C741bf7dee2e546df8d6782607df9deaa%7C0%7C0%7C638144113824006482%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=piAQoDtGBl1NyFHm1X5Kz2k5h3eHeH0v1e7WwiIPB6s%3D&reserved=0>, or unsubscribe<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAGKS3DPOIOSL2QI3FTJ75SLW4CSPDANCNFSM6AAAAAAVVMEIGA&data=05%7C01%7Cdotis%40usf.edu%7C197854ee262f4e7d90ae08db24b0c1db%7C741bf7dee2e546df8d6782607df9deaa%7C0%7C0%7C638144113824162690%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pYgyKrIzCvVzgYRkq9UeF7pDlb39O2U1DgMrmfXSWi4%3D&reserved=0>. You are receiving this because you were mentioned.Message ID: ***@***.***> [EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe.

[7yl4r]

Apparently not.

I don't know why we were getting that error. The error did go away after you reoriented, but the way it is now is obviously worse.

[dotis]

Ok. None of the other roi are done that way, so I will change these back and re-populate the ERDDAP and re-test. Thanks.

…

________________________________ From: Tylar ***@***.***> Sent: Tuesday, March 14, 2023 1:30 PM To: USF-IMARS/erddap-config ***@***.***> Cc: dotis ***@***.***>; Mention ***@***.***> Subject: Re: [USF-IMARS/erddap-config] moda_oc_7d_gom failing (Issue #35) Apparently not. I don't know why we were getting that error. The error did go away after you reoriented, but the way it is now is obviously worse. — Reply to this email directly, view it on GitHub<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FUSF-IMARS%2Ferddap-config%2Fissues%2F35%23issuecomment-1468529902&data=05%7C01%7Cdotis%40usf.edu%7C788f9a939c76462109e408db24b1d1f6%7C741bf7dee2e546df8d6782607df9deaa%7C0%7C0%7C638144118377070982%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5oqdlmLdYX7%2BQgmS4lqX3AcCV3jEFqM86JxYYU%2BB3%2Bs%3D&reserved=0>, or unsubscribe<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAGKS3DLS633RLZ4TYSODYODW4CTLPANCNFSM6AAAAAAVVMEIGA&data=05%7C01%7Cdotis%40usf.edu%7C788f9a939c76462109e408db24b1d1f6%7C741bf7dee2e546df8d6782607df9deaa%7C0%7C0%7C638144118377070982%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=s6vj4ya36fVx4vMy2O%2Bradfb7s9Cy4XT%2FL%2FTK4JpVgU%3D&reserved=0>. You are receiving this because you were mentioned.Message ID: ***@***.***> [EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe.