Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to Rename Dimensions with NcML (Works in 4.6) #490

Closed
rjdave opened this issue Apr 9, 2024 · 9 comments
Closed

Unable to Rename Dimensions with NcML (Works in 4.6) #490

rjdave opened this issue Apr 9, 2024 · 9 comments

Comments

@rjdave
Copy link

rjdave commented Apr 9, 2024

I'm running 2 identical catalogs, one on TDS 5.5-SNAPSHOT (downloaded 03/09/2024) and the other on THREDDS 4.6.20. I have a few datasets where I am renaming most of the variables and dimensions. This works fine in THREDDS 4.6.20, but renaming the dimensions in 5.5-SNAPSHOT results in the following error:

Error {
    code = 400;
    message = "NcML had fatal errors:NcML Dimension length is required ([Element: <dimension [Namespace: http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2]/>])
NcML Dimension length is required ([Element: <dimension [Namespace: http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2]/>])
NcML Dimension length is required ([Element: <dimension [Namespace: http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2]/>])
NcML Dimension length is required ([Element: <dimension [Namespace: http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2]/>])
";
};

If I remove the NcML that renames the dimensions, the aggregation works (with incorrect dimension names). Below is the catalog entry I'm using:

      <dataset    name="Atlantic/US East Coast Scraped"
                    ID="rtofs_us_east_scraped"
               urlPath="cool/rtofs/rtofs_us_east_scraped" >

         <metadata inherited="true">
            <documentation type="summary">
               Summary
            </documentation>
            <documentation type="source">
               ftp://ftp.ncei.noaa.gov/pub/data/nccf/com/rtofs/prod/
            </documentation>
            <documentation type="region">
               Atlantic/US East Coast
            </documentation>
            <documentation type="comment">
               Comment
            </documentation>
            <documentation type="region">
               atlantic_ocean_us_east_coast
            </documentation>
         </metadata>
         <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
            <aggregation dimName="MT" type="joinExisting">
               <scan location="/path/to/rtofs/"
                     suffix=".nc" olderThan="2 min" />
            </aggregation>
            <dimension name="time" orgName="MT" />
            <dimension name="y" orgName="Y" />
            <dimension name="x" orgName="X" />
            <dimension name="depth" orgName="Depth" />
            <variable name="time" orgName="MT" />
            <variable name="date" orgName="Date" />
            <variable name="depth" orgName="Depth">
               <attribute name="axis" value="z" />
            </variable>
            <variable name="y" orgName="Y">
               <attribute name="axis" value="y" />
            </variable>
            <variable name="x" orgName="X">
               <attribute name="axis" value="x" />
            </variable>
            <variable name="lat" orgName="Latitude" />
            <variable name="lon" orgName="Longitude" />
            <attribute name="region" value="Atlantic/US East Coast" />
            <attribute name="source" value="ftp://ftp.ncei.noaa.gov/pub/data/nccf/com/rtofs/prod/" />
            <attribute name="comment" value="Comment" />
         </netcdf>

      </dataset>
@tdrwenski
Copy link
Member

tdrwenski commented Apr 11, 2024 •
edited
Loading

Hi, I believe the variables may also need to be updated to understand their shape now depends on time instead of MT, since MT no longer exists. Something like:

         <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
            <aggregation dimName="MT" type="joinExisting">
               <scan location="/path/to/rtofs/"
                     suffix=".nc" olderThan="2 min" />
            </aggregation>
            <dimension name="time" orgName="MT" />
...
            <variable name="time" orgName="MT" />
            <variable name="time" shape="time" />
            <variable name="otherVariable" shape="time y x"/>
...
         </netcdf>

@rjdave
Copy link
Author

rjdave commented Apr 11, 2024

I had tried that in the past but tried again with addition of the 'shape' entries for each variable. It fails immediately with:

Error {
    code = 400;
    message = "Dimension time does not exist";
};

The failing catalog entry is now:

      <dataset    name="Atlantic/US East Coast Scraped"
                    ID="rtofs_us_east_scraped"
               urlPath="cool/rtofs/rtofs_us_east_scraped" >

         <metadata inherited="true">
            <documentation type="summary">
               Summary
            </documentation>
            <documentation type="source">
               ftp://ftp.ncei.noaa.gov/pub/data/nccf/com/rtofs/prod/
            </documentation>
            <documentation type="region">
               Atlantic/US East Coast
            </documentation>
            <documentation type="comment">
               Comment
            </documentation>
            <documentation type="region">
               atlantic_ocean_us_east_coast
            </documentation>
         </metadata>
         <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
            <aggregation dimName="time" type="joinExisting">
               <scan location="/path/to/rtofs/"
                     suffix=".nc" olderThan="2 min" />
            </aggregation>
            <dimension name="time" orgName="MT" />
            <dimension name="y" orgName="Y" />
            <dimension name="x" orgName="X" />
            <dimension name="depth" orgName="Depth" />
            <variable name="time" orgName="MT" />
            <variable name="time" shape="time" />
            <variable name="date" orgName="Date" />
            <variable name="date" shape="time" />
            <variable name="depth" orgName="Depth">
               <attribute name="axis" value="z" />
            </variable>
            <variable name="depth" shape="depth" />
            <variable name="y" orgName="Y">
               <attribute name="axis" value="y" />
            </variable>
            <variable name="y" shape="y" />
            <variable name="x" orgName="X">
               <attribute name="axis" value="x" />
            </variable>
            <variable name="x" shape="x" />
            <variable name="lat" orgName="Latitude" />
            <variable name="lat" shape="y x" />
            <variable name="lon" orgName="Longitude" />
            <variable name="lon" shape="y x" />
            <variable name="u" shape="time depth y x" />
            <variable name="v" shape="time depth y x" />
            <variable name="temperature" shape="time depth y x" />
            <variable name="salinity" shape="time depth y x" />
            <attribute name="region" value="Atlantic/US East Coast" />
            <attribute name="source" value="ftp://ftp.ncei.noaa.gov/pub/data/nccf/com/rtofs/prod/" />
            <attribute name="comment" value="Comment" />
         </netcdf>

      </dataset>

Out of curiosity, I changed the join 'dimName' back to MT and it still produced the same 'Dimension time does not exist' error (but not right away) so I removed all the lines with shape="time" and the error changed to 'Dimension depth does not exist' so I removed all the 'shape=' lines.

The exact catalog entry in the first post works in 4.6 with all the changed dimension and variable names appearing as expected. In 5.5-SNAPSHOT the dataset will load without the dimension renames. The variable renames work in 5.5-SNAPSHOT, albeit with the unchanged dimension names (MT, Y, X, and Depth).

@tdrwenski
Copy link
Member

I think the aggregation actually needs to be told the original (in the file) dimension, so <aggregation dimName="MT" type="joinExisting">, but all variables need to have their shapes updated for all of the renamed dimensions, so not just time but also depth, x, and y. I believe previously both the old and new versions of renamed dimensions existed in the dataset but now only the new version exists, so that's why you see the errors like 'Dimension depth does not exist'.

@rjdave
Copy link
Author

rjdave commented Apr 11, 2024

I tried that. I know it's a little hard to follow since, aside from 'MT' to 'time', all the dimension and variable renames are just a change in case (e.g. 'Depth' -> 'depth'). The case you suggesting is the case that resulted in the 'Dimension time does not exist' error that didn't happen right away (right before I removed all the shape="time" lines).

@tdrwenski tdrwenski mentioned this issue Apr 15, 2024
Merged
@tdrwenski
Copy link
Member

Hi @rjdave, we have a fix for this issue in the latest snapshot. Let us know if that resolves it for you!

@rjdave
Copy link
Author

rjdave commented Apr 16, 2024

Hi @tdrwenski, the latest snapshot fixed my issue. Thank you!

I did end up needing all those shape= entries for it to work, so I am pasting the dataset entry that works along with the ncdump -h of one of the raw files that is aggregated for future reference.

Catalog entry:

      <dataset    name="Atlantic/US East Coast Scraped"
                    ID="rtofs_us_east_scraped"
               urlPath="cool/rtofs/rtofs_us_east_scraped" >

         <metadata inherited="true">
            <documentation type="summary">
               Summary
            </documentation>
            <documentation type="source">
               ftp://ftp.ncei.noaa.gov/pub/data/nccf/com/rtofs/prod/
            </documentation>
            <documentation type="region">
               Atlantic/US East Coast
            </documentation>
            <documentation type="comment">
               Comment
            </documentation>
            <documentation type="region">
               atlantic_ocean_us_east_coast
            </documentation>
         </metadata>
         <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
            <aggregation dimName="MT" type="joinExisting">
               <scan location="/path/to/rtofs/"
                     suffix=".nc" olderThan="2 min" />
            </aggregation>
            <dimension name="time" orgName="MT" />
            <dimension name="y" orgName="Y" />
            <dimension name="x" orgName="X" />
            <dimension name="depth" orgName="Depth" />
            <variable name="time" orgName="MT" />
            <variable name="time" shape="time" />
            <variable name="date" orgName="Date" />
            <variable name="date" shape="time" />
            <variable name="depth" orgName="Depth">
               <attribute name="axis" value="z" />
            </variable>
            <variable name="depth" shape="depth" />
            <variable name="y" orgName="Y">
               <attribute name="axis" value="y" />
            </variable>
            <variable name="y" shape="y" />
            <variable name="x" orgName="X">
               <attribute name="axis" value="x" />
            </variable>
            <variable name="x" shape="x" />
            <variable name="lat" orgName="Latitude" />
            <variable name="lat" shape="y x" />
            <variable name="lon" orgName="Longitude" />
            <variable name="lon" shape="y x" />
            <variable name="u" shape="time depth y x" />
            <variable name="v" shape="time depth y x" />
            <variable name="temperature" shape="time depth y x" />
            <variable name="salinity" shape="time depth y x" />
            <attribute name="region" value="Atlantic/US East Coast" />
            <attribute name="source" value="ftp://ftp.ncei.noaa.gov/pub/data/nccf/com/rtofs/prod/" />
            <attribute name="comment" value="Comment" />
         </netcdf>

      </dataset>

ncdump -h of one of the aggregated NetCDF files:

netcdf rtofs_glo_3dz_f006_6hrly_hvr_US_east {
dimensions:
        MT = UNLIMITED ; // (1 currently)
        Y = 1710 ;
        X = 742 ;
        Depth = 40 ;
variables:
        double MT(MT) ;
                MT:long_name = "time" ;
                MT:units = "days since 1900-12-31 00:00:00" ;
                MT:calendar = "standard" ;
                MT:axis = "T" ;
        double Date(MT) ;
                Date:long_name = "date" ;
                Date:units = "day as %Y%m%d.%f" ;
                Date:C_format = "%13.4f" ;
                Date:FORTRAN_format = "(f13.4)" ;
        float Depth(Depth) ;
                Depth:standard_name = "depth" ;
                Depth:units = "m" ;
                Depth:positive = "down" ;
                Depth:axis = "Z" ;
        int Y(Y) ;
                Y:point_spacing = "even" ;
                Y:axis = "Y" ;
        int X(X) ;
                X:point_spacing = "even" ;
                X:axis = "X" ;
        float Latitude(Y, X) ;
                Latitude:standard_name = "latitude" ;
                Latitude:units = "degrees_north" ;
        float Longitude(Y, X) ;
                Longitude:standard_name = "longitude" ;
                Longitude:units = "degrees_east" ;
        float u(MT, Depth, Y, X) ;
                u:coordinates = "Longitude Latitude Date" ;
                u:standard_name = "eastward_sea_water_velocity" ;
                u:units = "m/s" ;
                u:_FillValue = 1.267651e+30f ;
                u:valid_range = -1.847417f, 2.467955f ;
                u:long_name = " u-veloc. [92.8H]" ;
        float v(MT, Depth, Y, X) ;
                v:coordinates = "Longitude Latitude Date" ;
                v:standard_name = "northward_sea_water_velocity" ;
                v:units = "m/s" ;
                v:_FillValue = 1.267651e+30f ;
                v:valid_range = -1.832807f, 2.062172f ;
                v:long_name = " v-veloc. [92.8H]" ;
        float temperature(MT, Depth, Y, X) ;
                temperature:coordinates = "Longitude Latitude Date" ;
                temperature:standard_name = "sea_water_potential_temperature" ;
                temperature:units = "degC" ;
                temperature:_FillValue = 1.267651e+30f ;
                temperature:valid_range = -8.505487f, 32.26008f ;
                temperature:long_name = "  temp [92.8H]" ;
        float salinity(MT, Depth, Y, X) ;
                salinity:coordinates = "Longitude Latitude Date" ;
                salinity:standard_name = "sea_water_salinity" ;
                salinity:units = "psu" ;
                salinity:_FillValue = 1.267651e+30f ;
                salinity:valid_range = 0.3615089f, 40.01632f ;
                salinity:long_name = " salinity [92.8H]" ;

// global attributes:
                :Conventions = "CF-1.0" ;
                :title = "HYCOM ATLb2.00" ;
                :institution = "National Centers for Environmental Prediction" ;
                :source = "HYCOM archive file" ;
                :experiment = "92.8" ;
                :history = "archv2ncdf3z" ;
}

@rjdave rjdave closed this as completed Apr 16, 2024
@rjdave
Copy link
Author

rjdave commented Jun 13, 2024

So this isn't actually fixed, it just displays the data form correctly. If you try to retrieve any actual data there are errors in the browser (depth is renamed from Depth):

Error {
    code = 500;
    message = "null";
};

and the logs:

2024-06-13T18:35:05.352 +0000 [ 175069397][   45908] INFO  - threddsServlet - Remote host: 74.102.8.5 - Request: "GET /thredds/dodsC/cool/rtofs/rtofs_us_east_scraped.ascii?depth[0:1:39] HTTP/1.1"
2024-06-13T18:35:09.252 +0000 [ 175073297][   45908] ERROR - thredds.server.opendap.OpendapServlet - request= ReqState:
  serverClassName:    'null'
  dataSet:            'cool/rtofs/rtofs_us_east_scraped'
  requestSuffix:      'ascii'
  CE:                 'depth[0:1:39]'
  compressOK:          true
  InitParameters:

java.lang.NullPointerException: null
   at ucar.nc2.internal.ncml.AggProxyReader.reallyRead(AggProxyReader.java:37) ~[cdm-core-5.5.4-SNAPSHOT.jar:5.5.4-SNAPSHOT]
   at ucar.nc2.dataset.VariableDS._read(VariableDS.java:470) ~[cdm-core-5.5.4-SNAPSHOT.jar:5.5.4-SNAPSHOT]
   at ucar.nc2.dataset.VariableDS._read(VariableDS.java:481) ~[cdm-core-5.5.4-SNAPSHOT.jar:5.5.4-SNAPSHOT]
   at ucar.nc2.Variable.read(Variable.java:600) ~[cdm-core-5.5.4-SNAPSHOT.jar:5.5.4-SNAPSHOT]
   at ucar.nc2.Variable.read(Variable.java:574) ~[cdm-core-5.5.4-SNAPSHOT.jar:5.5.4-SNAPSHOT]
   at thredds.server.opendap.NcSDArray.read(NcSDArray.java:85) ~[classes/:5.5-SNAPSHOT]
   at opendap.servlet.AsciiWriter.writeAsc(AsciiWriter.java:76) ~[opendap-servlet-5.5-SNAPSHOT.jar:5.5-SNAPSHOT]
   at opendap.servlet.AsciiWriter.toASCII(AsciiWriter.java:38) ~[opendap-servlet-5.5-SNAPSHOT.jar:5.5-SNAPSHOT]
   at thredds.server.opendap.OpendapServlet.doGetASC(OpendapServlet.java:286) ~[classes/:5.5-SNAPSHOT]
   at thredds.server.opendap.OpendapServlet.doGet(OpendapServlet.java:184) [classes/:5.5-SNAPSHOT]
   at jdk.internal.reflect.GeneratedMethodAccessor103.invoke(Unknown Source) ~[?:?]
   at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
   at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
   at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-5.3.34.jar:5.3.34]
   at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) [spring-web-5.3.34.jar:5.3.34]
   at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) [spring-webmvc-5.3.34.jar:5.3.34]
   at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:903) [spring-webmvc-5.3.34.jar:5.3.34]
   at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:809) [spring-webmvc-5.3.34.jar:5.3.34]
   at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) [spring-webmvc-5.3.34.jar:5.3.34]
   at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072) [spring-webmvc-5.3.34.jar:5.3.34]
   at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965) [spring-webmvc-5.3.34.jar:5.3.34]
   at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) [spring-webmvc-5.3.34.jar:5.3.34]
   at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) [spring-webmvc-5.3.34.jar:5.3.34]
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:489) [servlet-api.jar:?]
   at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) [spring-webmvc-5.3.34.jar:5.3.34]
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:583) [servlet-api.jar:?]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:212) [catalina.jar:8.5.100]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:156) [catalina.jar:8.5.100]
   at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) [tomcat-websocket.jar:8.5.100]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:181) [catalina.jar:8.5.100]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:156) [catalina.jar:8.5.100]
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:337) [spring-security-web-5.7.12.jar:5.7.12]
   at thredds.servlet.filter.RequestBracketingLogMessageFilter.doFilter(RequestBracketingLogMessageFilter.java:50) [classes/:5.5-SNAPSHOT]
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.12.jar:5.7.12]
   at thredds.servlet.filter.RequestQueryFilter.doFilter(RequestQueryFilter.java:90) [classes/:5.5-SNAPSHOT]
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.12.jar:5.7.12]
   at thredds.servlet.filter.HttpHeadFilter.doFilter(HttpHeadFilter.java:47) [classes/:5.5-SNAPSHOT]
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.12.jar:5.7.12]
   at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:221) [spring-security-web-5.7.12.jar:5.7.12]
   at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186) [spring-security-web-5.7.12.jar:5.7.12]
   at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) [spring-web-5.3.34.jar:5.3.34]
   at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) [spring-web-5.3.34.jar:5.3.34]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:181) [catalina.jar:8.5.100]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:156) [catalina.jar:8.5.100]
   at org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:71) [log4j-web-2.17.1.jar:2.17.1]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:181) [catalina.jar:8.5.100]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:156) [catalina.jar:8.5.100]
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168) [catalina.jar:8.5.100]
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) [catalina.jar:8.5.100]
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483) [catalina.jar:8.5.100]
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) [catalina.jar:8.5.100]
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) [catalina.jar:8.5.100]
   at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:761) [catalina.jar:8.5.100]
   at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:679) [catalina.jar:8.5.100]
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [catalina.jar:8.5.100]
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346) [catalina.jar:8.5.100]
   at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:617) [tomcat-coyote.jar:8.5.100]
   at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) [tomcat-coyote.jar:8.5.100]
   at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:934) [tomcat-coyote.jar:8.5.100]
   at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1690) [tomcat-coyote.jar:8.5.100]
   at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) [tomcat-coyote.jar:8.5.100]
   at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) [tomcat-util.jar:8.5.100]
   at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-util.jar:8.5.100]
   at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) [tomcat-util.jar:8.5.100]
   at java.lang.Thread.run(Thread.java:829) [?:?]
2024-06-13T18:35:09.253 +0000 [ 175073298][   45908] INFO  - threddsServlet - Request Completed - 500 - -1 - 3901

It will however output time (renamed from MT). If you request date (renamed from Date) you get the error:

Error {
    code = 400;
    message = "Variable 'date' does not exist in aggregation.";
};

and this in the logs:

2024-06-13T18:37:44.783 +0000 [ 175228828][   45946] INFO  - threddsServlet - Remote host: 74.102.8.5 - Request: "GET /thredds/dodsC/cool/rtofs/rtofs_us_east_scraped.ascii?date%5B0:1:7%5D HTTP/1.1"
2024-06-13T18:37:45.355 +0000 [ 175229400][   45946] ERROR - ucar.nc2.internal.ncml.Aggregation - AggOuterDimension can't find date in /home/hurricaneadm/data/rtofs/rtofs.20190717/rtofs_glo_3dz_f006_6hrly_hvr_US_east.nc
2024-06-13T18:37:45.365 +0000 [ 175229410][   45946] INFO  - thredds.server.opendap.OpendapServlet - request= ReqState:
  serverClassName:    'null'
  dataSet:            'cool/rtofs/rtofs_us_east_scraped'
  requestSuffix:      'ascii'
  CE:                 'date[0:1:7]'
  compressOK:          true
  InitParameters:

java.lang.IllegalArgumentException: Variable 'date' does not exist in aggregation.
   at ucar.nc2.internal.ncml.AggDatasetOuter.read(AggDatasetOuter.java:289) ~[cdm-core-5.5.4-SNAPSHOT.jar:5.5.4-SNAPSHOT]
   at ucar.nc2.internal.ncml.AggregationOuter.reallyRead(AggregationOuter.java:421) ~[cdm-core-5.5.4-SNAPSHOT.jar:5.5.4-SNAPSHOT]
   at ucar.nc2.dataset.VariableDS._read(VariableDS.java:488) ~[cdm-core-5.5.4-SNAPSHOT.jar:5.5.4-SNAPSHOT]
   at ucar.nc2.Variable.read(Variable.java:600) ~[cdm-core-5.5.4-SNAPSHOT.jar:5.5.4-SNAPSHOT]
   at ucar.nc2.Variable.read(Variable.java:574) ~[cdm-core-5.5.4-SNAPSHOT.jar:5.5.4-SNAPSHOT]
   at thredds.server.opendap.NcSDArray.read(NcSDArray.java:85) ~[classes/:5.5-SNAPSHOT]
   at opendap.servlet.AsciiWriter.writeAsc(AsciiWriter.java:76) ~[opendap-servlet-5.5-SNAPSHOT.jar:5.5-SNAPSHOT]
   at opendap.servlet.AsciiWriter.toASCII(AsciiWriter.java:38) ~[opendap-servlet-5.5-SNAPSHOT.jar:5.5-SNAPSHOT]
   at thredds.server.opendap.OpendapServlet.doGetASC(OpendapServlet.java:286) ~[classes/:5.5-SNAPSHOT]
   at thredds.server.opendap.OpendapServlet.doGet(OpendapServlet.java:184) [classes/:5.5-SNAPSHOT]
   at jdk.internal.reflect.GeneratedMethodAccessor103.invoke(Unknown Source) ~[?:?]
   at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
   at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
   at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-5.3.34.jar:5.3.34]
   at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) [spring-web-5.3.34.jar:5.3.34]
   at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) [spring-webmvc-5.3.34.jar:5.3.34]
   at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:903) [spring-webmvc-5.3.34.jar:5.3.34]
   at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:809) [spring-webmvc-5.3.34.jar:5.3.34]
   at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) [spring-webmvc-5.3.34.jar:5.3.34]
   at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072) [spring-webmvc-5.3.34.jar:5.3.34]
   at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965) [spring-webmvc-5.3.34.jar:5.3.34]
   at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) [spring-webmvc-5.3.34.jar:5.3.34]
   at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) [spring-webmvc-5.3.34.jar:5.3.34]
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:489) [servlet-api.jar:?]
   at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) [spring-webmvc-5.3.34.jar:5.3.34]
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:583) [servlet-api.jar:?]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:212) [catalina.jar:8.5.100]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:156) [catalina.jar:8.5.100]
   at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) [tomcat-websocket.jar:8.5.100]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:181) [catalina.jar:8.5.100]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:156) [catalina.jar:8.5.100]
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:337) [spring-security-web-5.7.12.jar:5.7.12]
   at thredds.servlet.filter.RequestBracketingLogMessageFilter.doFilter(RequestBracketingLogMessageFilter.java:50) [classes/:5.5-SNAPSHOT]
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.12.jar:5.7.12]
   at thredds.servlet.filter.RequestQueryFilter.doFilter(RequestQueryFilter.java:90) [classes/:5.5-SNAPSHOT]
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.12.jar:5.7.12]
   at thredds.servlet.filter.HttpHeadFilter.doFilter(HttpHeadFilter.java:47) [classes/:5.5-SNAPSHOT]
   at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) [spring-security-web-5.7.12.jar:5.7.12]
   at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:221) [spring-security-web-5.7.12.jar:5.7.12]
   at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186) [spring-security-web-5.7.12.jar:5.7.12]
   at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) [spring-web-5.3.34.jar:5.3.34]
   at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) [spring-web-5.3.34.jar:5.3.34]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:181) [catalina.jar:8.5.100]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:156) [catalina.jar:8.5.100]
   at org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:71) [log4j-web-2.17.1.jar:2.17.1]
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:181) [catalina.jar:8.5.100]
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:156) [catalina.jar:8.5.100]
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168) [catalina.jar:8.5.100]
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) [catalina.jar:8.5.100]
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483) [catalina.jar:8.5.100]
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) [catalina.jar:8.5.100]
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) [catalina.jar:8.5.100]
   at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:761) [catalina.jar:8.5.100]
   at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:679) [catalina.jar:8.5.100]
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [catalina.jar:8.5.100]
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346) [catalina.jar:8.5.100]
   at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:617) [tomcat-coyote.jar:8.5.100]
   at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) [tomcat-coyote.jar:8.5.100]
   at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:934) [tomcat-coyote.jar:8.5.100]
   at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1690) [tomcat-coyote.jar:8.5.100]
   at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) [tomcat-coyote.jar:8.5.100]
   at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) [tomcat-util.jar:8.5.100]
   at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-util.jar:8.5.100]
   at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) [tomcat-util.jar:8.5.100]
   at java.lang.Thread.run(Thread.java:829) [?:?]
2024-06-13T18:37:45.366 +0000 [ 175229411][   45946] INFO  - threddsServlet - Request Completed - 400 - -1 - 583

@rjdave rjdave reopened this Jun 13, 2024
@tdrwenski
Copy link
Member

Can you try doing the rename and the reshape in the same xml element? for me this seems to work correctly:

            <variable name="date" orgName="Date" shape="time"/>

instead of

            <variable name="date" orgName="Date" />
            <variable name="date" shape="time" />

I believe in the latter case the second "date" is overwriting the first and the "orgName" is not getting saved.

@rjdave
Copy link
Author

rjdave commented Jun 17, 2024

Yes, that works! Thanks for the quick reply and easy fix.

@rjdave rjdave closed this as completed Jun 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rjdave @tdrwenski