Geonetwork 3 integration #1120

Closed
landryb opened this Issue Nov 12, 2015 · 34 comments

Projects

None yet

3 participants

@landryb
Member
landryb commented Nov 12, 2015

Tracking bug - right now @pmauduit has started working in:

I've started testing this at runtime on an instance of master using debian packages (generated from the above branch) and georchestra datadir, and so far it starts and displays fine.

Found some non-georchestra issues (geonetwork/core-geonetwork#1294 being the most annoying right now, but there's also geonetwork/core-geonetwork#1288, geonetwork/core-geonetwork#1295, geonetwork/core-geonetwork#1293) and only a small annoyance with ldap attributes mapping to user fields, solved by this hack in config-security-georchestra.xml - https://pastebin.mozilla.org/8851981

Big issue will still be the migration of existing databases/instances, since the geonetwork database migration is not... really reliable. We still have to figure out a way to let the users decide whether they want to keep their old gn 2.11 for now, but new installs should default to gn3.

summary

@fvanderbiest
Member

new installs should default to gn3

+1

@fvanderbiest fvanderbiest added this to the 15.12 milestone Nov 12, 2015
@pmauduit
Member

new installs should default to gn3

👍 for me as well

@landryb
Member
landryb commented Nov 13, 2015

When editing a metadata, the editor toolbar is overlayed on top of georchestra header, see http://i.imgur.com/US3oGsa.png. To be worked on.....

@landryb
Member
landryb commented Nov 13, 2015

When one is on the internal ol3 viewer, its sized according to the full page size without taking into account the header height, and thus there's a scrollbar...

@landryb
Member
landryb commented Nov 13, 2015

With the default templates in the datadir, when going on a metadata, switching to 'full view' renders a blank content (after the georchestra & geonetwork headers) complaining about missing groovy classes ?

Caused by: java.lang.AssertionError: Errors occurred while compiling files:

/data/webapps/geonetwork/config/schema_plugins/iso19139/formatter/groovy/iso19139/Handlers.groovy:
startup failed:
file:/data/webapps/geonetwork/config/schema_plugins/iso19139/formatter/groovy/iso19139/Handlers.groovy: 12: unable to resolve class common.Handlers 
 @ line 12, column 5.
       common.Handlers commonHandlers
       ^
@landryb
Member
landryb commented Nov 16, 2015

Setting loglevel to INFO gives more details on the "broken views"

2015-11-16 11:47:03,916 INFO  [geonetwork] - Initializing the Formatter with id: full_view
2015-11-16 11:47:07,162 INFO  [geonetwork] - Error while initializing the Formatter with id: full_view
java.lang.AssertionError: Errors occurred while compiling files:
/data/webapps/geonetwork/config/schema_plugins/iso19139/formatter/groovy/iso19139/Handlers.groovy:
startup failed:
file:/data/webapps/geonetwork/config/schema_plugins/iso19139/formatter/groovy/iso19139/Handlers.groovy: 12: unable to resolve class common.Handlers
 @ line 12, column 5.
       common.Handlers commonHandlers

        at org.fao.geonet.services.metadata.format.GroovyFormatter.loadScripts(GroovyFormatter.java:200)
        at org.fao.geonet.services.metadata.format.GroovyFormatter.getParentClassLoader(GroovyFormatter.java:143)
        at org.fao.geonet.services.metadata.format.GroovyFormatter.getParentClassLoader(GroovyFormatter.java:129)
        at org.fao.geonet.services.metadata.format.GroovyFormatter.createTransformer(GroovyFormatter.java:89)
        at org.fao.geonet.services.metadata.format.GroovyFormatter.format(GroovyFormatter.java:71)
        at org.fao.geonet.services.metadata.format.Format$FormatMetadata.call(Format.java:644)
        at org.fao.geonet.services.metadata.format.Format$FormatMetadata.call(Format.java:625)
        at org.fao.geonet.services.metadata.format.cache.FormatterCache.get(FormatterCache.java:150)
        at org.fao.geonet.services.metadata.format.Format.exec(Format.java:358)
        at org.fao.geonet.Geonetwork$1.run(Geonetwork.java:492)
        at java.lang.Thread.run(Thread.java:745)
2015-11-16 11:47:07,164 INFO  [geonetwork] - Initializing the Formatter with id: xml_view
2015-11-16 11:47:07,203 INFO  [geonetwork] - Error while initializing the Formatter with id: xml_view
BadParameterEx : Format bundle xml_view does not exist is not a valid value for: xsl
        at org.fao.geonet.services.metadata.format.AbstractFormatService.getAndVerifyFormatDir(AbstractFormatService.java:93)
        at org.fao.geonet.services.metadata.format.Format.createFormatterAndParams(Format.java:441)
        at org.fao.geonet.services.metadata.format.Format.loadMetadataAndCreateFormatterAndParams(Format.java:421)
        at org.fao.geonet.services.metadata.format.Format$FormatMetadata.call(Format.java:640)
        at org.fao.geonet.services.metadata.format.Format$FormatMetadata.call(Format.java:625)
        at org.fao.geonet.services.metadata.format.cache.FormatterCache.get(FormatterCache.java:150)
        at org.fao.geonet.services.metadata.format.Format.exec(Format.java:358)
        at org.fao.geonet.Geonetwork$1.run(Geonetwork.java:492)
        at java.lang.Thread.run(Thread.java:745)
2015-11-16 11:47:07,203 INFO  [geonetwork] - Initializing the Formatter with id: hierarchy_view
2015-11-16 11:47:07,234 INFO  [geonetwork] - Error while initializing the Formatter with id: hierarchy_view
BadParameterEx : Format bundle hierarchy_view does not exist is not a valid value for: xsl
        at org.fao.geonet.services.metadata.format.AbstractFormatService.getAndVerifyFormatDir(AbstractFormatService.java:93)
        at org.fao.geonet.services.metadata.format.Format.createFormatterAndParams(Format.java:441)
        at org.fao.geonet.services.metadata.format.Format.loadMetadataAndCreateFormatterAndParams(Format.java:421)
        at org.fao.geonet.services.metadata.format.Format$FormatMetadata.call(Format.java:640)
        at org.fao.geonet.services.metadata.format.Format$FormatMetadata.call(Format.java:625)
        at org.fao.geonet.services.metadata.format.cache.FormatterCache.get(FormatterCache.java:129)
        at org.fao.geonet.services.metadata.format.Format.exec(Format.java:358)
        at org.fao.geonet.Geonetwork$1.run(Geonetwork.java:492)
        at java.lang.Thread.run(Thread.java:745)
@landryb
Member
landryb commented Nov 16, 2015

For this last issue, i've found the source of the issue : $geonetwork.datadir/data/formatter is not versionned yet/empty in the minimal datadir repo (https://github.com/georchestra/geonetwork_minimal_datadir/tree/gn3.0.x), while there should be stuff coming from https://github.com/geonetwork/core-geonetwork/tree/develop/web/src/main/webapp/WEB-INF/data/data/formatter.

If i copy the whole content into my formatter datadir, the xml_view is properly initialized (but not available in the UI, even if enabled in settings), while full_view and hierarchy_view are still broken but at least there should be less to do.

2015-11-16 15:53:13,909 INFO  [geonetwork] - Initializing the Formatter with id: full_view
2015-11-16 15:53:19,088 INFO  [geonetwork] - Error while initializing the Formatter with id: full_view
org.fao.geonet.services.metadata.format.groovy.template.TemplateException: There is no object in the model map with the id 'item' in the model map.  The model selection expression is: item.name'.
The current options are:
[abstract, logo, keywords, navBar, isPDF, links, navBarOverflow, formats, extents, content, isHTML, title, thumbnail, addOverviewNavItem, associated, pageTitle, showNavOverflow, addCompleteNavItem]
        at org.fao.geonet.services.metadata.format.groovy.template.TRenderContext.getModelValue(TRenderContext.java:112)
        at org.fao.geonet.services.metadata.format.groovy.template.TRenderContext.getModelValue(TRenderContext.java:95)
        at org.fao.geonet.services.metadata.format.groovy.template.TextContentReplacement.text(TextContentReplacement.java:21)
        at org.fao.geonet.services.metadata.format.groovy.template.TextBlock.render(TextBlock.java:23)
        at org.fao.geonet.services.metadata.format.groovy.template.TNodeTextContent.render(TNodeTextContent.java:21)
        at org.fao.geonet.services.metadata.format.groovy.template.FileResult.toString(FileResult.java:29)
        at java.lang.String.valueOf(String.java:2849)
        at java.lang.StringBuilder.append(StringBuilder.java:128)
        at org.fao.geonet.services.metadata.format.groovy.Handler.processResult(Handler.java:75)
        at org.fao.geonet.services.metadata.format.groovy.Handler.handle(Handler.java:55)
        at org.fao.geonet.services.metadata.format.groovy.TransformEngine.processElement(TransformEngine.java:76)
        at org.fao.geonet.services.metadata.format.groovy.Transformer.processRoot(Transformer.java:86)
        at org.fao.geonet.services.metadata.format.groovy.Transformer.apply(Transformer.java:54)
        at org.fao.geonet.services.metadata.format.GroovyFormatter.format(GroovyFormatter.java:76)
        at org.fao.geonet.services.metadata.format.Format$FormatMetadata.call(Format.java:644)
        at org.fao.geonet.services.metadata.format.Format$FormatMetadata.call(Format.java:625)
        at org.fao.geonet.services.metadata.format.cache.FormatterCache.get(FormatterCache.java:150)
        at org.fao.geonet.services.metadata.format.Format.exec(Format.java:358)
        at org.fao.geonet.Geonetwork$1.run(Geonetwork.java:492)
        at java.lang.Thread.run(Thread.java:745)
2015-11-16 15:53:19,093 INFO  [geonetwork] - Initializing the Formatter with id: xml_view
2015-11-16 15:53:19,595 INFO  [geonetwork] - Initializing the Formatter with id: hierarchy_view
2015-11-16 15:53:19,719 INFO  [geonetwork.datamanager] - XLink Resolver enabled.
2015-11-16 15:53:20,995 INFO  [geonetwork] - Error while initializing the Formatter with id: hierarchy_view
org.fao.geonet.services.metadata.format.groovy.template.TemplateException: There is no object in the model map with the id 'linkBlock' in the model map.  The model selection expression is: linkBlock.name'.
The current options are:
[associated]
        at org.fao.geonet.services.metadata.format.groovy.template.TRenderContext.getModelValue(TRenderContext.java:112)
        at org.fao.geonet.services.metadata.format.groovy.template.TRenderContext.getModelValue(TRenderContext.java:95)
        at org.fao.geonet.services.metadata.format.groovy.template.TextContentReplacement.text(TextContentReplacement.java:21)
        at org.fao.geonet.services.metadata.format.groovy.template.TextBlock.render(TextBlock.java:23)
        at org.fao.geonet.services.metadata.format.groovy.template.TNodeTextContent.render(TNodeTextContent.java:21)
        at org.fao.geonet.services.metadata.format.groovy.template.FileResult.toString(FileResult.java:29)
        at java.lang.String.valueOf(String.java:2849)
        at java.lang.StringBuilder.append(StringBuilder.java:128)
        at org.fao.geonet.services.metadata.format.groovy.Handler.processResult(Handler.java:75)
        at org.fao.geonet.services.metadata.format.groovy.Handler.handle(Handler.java:55)
        at org.fao.geonet.services.metadata.format.groovy.TransformEngine.processElement(TransformEngine.java:76)
        at org.fao.geonet.services.metadata.format.groovy.Transformer.processRoot(Transformer.java:86)
        at org.fao.geonet.services.metadata.format.groovy.Transformer.apply(Transformer.java:54)
        at org.fao.geonet.services.metadata.format.GroovyFormatter.format(GroovyFormatter.java:76)
        at org.fao.geonet.services.metadata.format.Format$FormatMetadata.call(Format.java:644)
        at org.fao.geonet.services.metadata.format.Format$FormatMetadata.call(Format.java:625)
        at org.fao.geonet.services.metadata.format.cache.FormatterCache.get(FormatterCache.java:129)
        at org.fao.geonet.services.metadata.format.Format.exec(Format.java:358)
        at org.fao.geonet.Geonetwork$1.run(Geonetwork.java:492)
        at java.lang.Thread.run(Thread.java:745)
@landryb
Member
landryb commented Nov 16, 2015

Same thing for data/resources/map, should be in the default datadir.

@landryb
Member
landryb commented Nov 17, 2015

geonetwork/core-geonetwork#1302, geonetwork/core-geonetwork#1301 & geonetwork/core-geonetwork#1300 could be merged too until they're merged upstream. I'll have a look at the group name / translation issue (#1294)

@pmauduit
Member

The 4 PR above have been merged, my gn3 branch is realigned, waiting for the CI to test the whole thing.

@landryb
Member
landryb commented Nov 20, 2015

Since the pastebin in the first comment expired, i'm adding again the 'hack' i use to customize/fix the ldap attributes mapping in the georchestra datadir, otherwise they're overriden by config-security-overrides.properties coming from the webapp, and if unset no default mapping is used, leading to a crash at startup. In a normal situation, the ldapUserContextMapper.mapping found in geonetwork/geonetwork.properties should apply...

--- a/geonetwork/config/config-security-georchestra.xml
+++ b/geonetwork/config/config-security-georchestra.xml
@@ -39,7 +39,9 @@

   <!-- config-security-ldap.xml customized for geOrchestra -->

+<!--
   <ctx:property-override location="WEB-INF/config-security/config-security-overrides.properties" order="2"/>
+-->
   <!-- using the one from the geOrchestra datadir first (if available) -->
   <ctx:property-placeholder location="file:${georchestra.datadir}/geonetwork/geonetwork.properties" ignore-resource-not-found="true" ignore-unresolvable="true" order="1"/>
   <!-- Add ldap authentication to authentication manager -->
@@ -74,7 +76,21 @@
         location and the extraction pattern. -->
   <bean id="ldapUserContextMapper" class="org.fao.geonet.kernel.security.ldap.LDAPUserDetailsContextMapperWithProfileSearch">
     <property name="mapping">
-      <map/>
+      <map>
+        <entry key="name" value="givenName,NoFirstName"/>
+        <entry key="surname" value="sn,NoName"/>
+        <entry key="mail" value="mail,me@organization"/>
+        <entry key="organisation" value="o,myorganization"/>
+        <entry key="kind" value="title,"/>
+        <entry key="address" value=","/>
+        <entry key="zip" value=","/>
+        <entry key="state" value=","/>
+        <entry key="city" value=","/>
+        <entry key="country" value=","/>
+        <entry key="privilege" value=","/>
+        <entry key="phone" value="telephoneNumber,"/>
+        <entry key="profile" value=",RegisteredUser"/>
+      </map>

@pmauduit
Member

TODO: Adding a ctx:property-override pointing to the geonetwork.properties from the datadir. the property-override is "smarter" than property-placeholder when it deals to bean configuration.

@pmauduit pmauduit added a commit to pmauduit/core-geonetwork that referenced this issue Nov 20, 2015
@pmauduit pmauduit security-overrides in its own file + sourcing it in the xml
See
georchestra/georchestra#1120 (comment)
for the motivation behind this commit.

Tests: Runtime
2cbdcc6
@landryb
Member
landryb commented Nov 23, 2015

Other runtime issue - on the main geonetwork ol3/ngeo viewer, if you try to load a wms layer from the included georchestra geoserver, you get a 403 from the geonetwork proxy - while i think it shouldnt be used..

http://georchestra.craig.fr/geocat/proxy?url=http%3A%2F%2Fgeorchestra.craig.fr%2Fwxs%2Fpublic%2Fwms%3Fservice%3DWMS%26request%3DGetCapabilities

The getcap request doesnt even reach the sec-proxy/cas.

It works without issues when querying another 'open' mapserver/geoserver outside of georchestra.

@fvanderbiest
Member

Yes, the local proxy should not be used, but it should work ...

Looks like our security proxy is really catching too many user agents.
We need someone to help us with #960, ideally for 15.12

@landryb
Member
landryb commented Nov 24, 2015

The behaviour is still the same after commenting out all the entries in ${georchestra.datadir}/security-proxy/user-agents.properties - and according to the default config in https://github.com/geonetwork/core-geonetwork/blob/develop/web/src/main/webResources/WEB-INF/web.xml#L330 the proxy doesnt 'limit' targets by default, unless i misunderstand this config snippet.

@landryb
Member
landryb commented Nov 24, 2015

Should also note that the 403 is the "Désolé, vous n'avez pas accès à cette page." message/html page coming from geor with the header included, and not one of the messages/exceptions from https://github.com/geonetwork/core-geonetwork/blob/develop/core/src/main/java/org/geonetwork/http/proxy/HttpProxyServlet.java

@landryb
Member
landryb commented Nov 24, 2015

Oookay, i think i have a final explanation for this geonetwork proxy not working, and a config fix...

This last 403 error was due to a misconfiguration on my side - the fqdn is ip-filtered, and since the requests coming from the GN proxy are from the backend IP, they weren't allowed. After fixing the list of whitelisted IPs in my reverse-proxy & restarting again proxycas tomcat (to ensure the Apache-HttpClient UA wasnt 401-challenged) - i still get a 401 code :(

So from what i understand, the UA filtering is still trigerred by the list in https://github.com/georchestra/georchestra/blob/master/security-proxy/src/main/webapp/WEB-INF/applicationContext-security.xml#L79-L87 because all values in ${georchestra.datadir}/security-proxy/user-agents.properties are commented out.

If i read the code in https://github.com/georchestra/georchestra/blob/master/security-proxy/src/main/java/org/georchestra/security/BasicAuthChallengeByUserAgent.java#L66 that confirms it - my file is not empty but contains no 'valid entries' so _userAgents.clear() isnt called - so the default UA list comes from applicationContext-security.xml. Adding a "dummy" entry useragent0.value=.*nonexistentUA.* to ${georchestra.datadir}/security-proxy/user-agents.properties workarounds the issue and i can load that damn getcapabilities in Gn3 default viewer. Phew.

So maybe, for this specific issue.... in BasicAuthChallengeByUserAgent.java:init() the case when user-agents.properties is found but empty should be handled as 'i dont want any UA filtering to happen' and _userAgents.clear() should be called ?

@landryb
Member
landryb commented Nov 26, 2015

In some circumstances, gn3 doesnt properly 'see' the user as authenticated and triggers srv/eng/catalog.signin service, which prompts for login/password, which for some reason doesnt properly auth the user with cas credentials - in georchestra context, either this should properly integrate with cas and recheck the ticket or the request should redirect to the cas login prompt...

@landryb
Member
landryb commented Nov 26, 2015

Quite critical issue to track in the integration too : geonetwork/core-geonetwork#1015 (comment) - to me right now it's a serious security issue upstream.

@landryb
Member
landryb commented Nov 27, 2015

Updated to last commit in gn3 branch, webapp fails to initialize wro4j cache. The xml services work, but the UI is 100% missing. Traceback at startup:

2015-11-27 16:35:36,093 ERROR [geonetwork.wro4j] - Error while loading wro4j model
java.lang.IllegalArgumentException: Parameter 'directory' is not a directory
        at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:460)
        at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:546)
        at org.apache.commons.io.FileUtils.iterateFiles(FileUtils.java:565)
        at org.fao.geonet.wro4j.GeonetWroModelFactory$ResourceDesc$1.iterator(GeonetWroModelFactory.java:707)
        at org.fao.geonet.wro4j.GeonetWroModelFactory.addCssGroupsByRequireDependencies(GeonetWroModelFactory.java:464)
        at org.fao.geonet.wro4j.GeonetWroModelFactory.loadGroupsUsingRequireDependencyManagement(GeonetWroModelFactory.java:395)
        at org.fao.geonet.wro4j.GeonetWroModelFactory.createModel(GeonetWroModelFactory.java:191)
        at org.fao.geonet.wro4j.GeonetWroModelFactory.createModel(GeonetWroModelFactory.java:163)
        at org.fao.geonet.wro4j.GeonetWroModelFactory.create(GeonetWroModelFactory.java:108)
        at org.fao.geonet.wro4j.GeonetWroModelFactory.create(GeonetWroModelFactory.java:65)
        at ro.isdc.wro.model.factory.WroModelFactoryDecorator.create(WroModelFactoryDecorator.java:27)
        at ro.isdc.wro.model.factory.FallbackAwareWroModelFactory.create(FallbackAwareWroModelFactory.java:48)
        at ro.isdc.wro.model.factory.FallbackAwareWroModelFactory.create(FallbackAwareWroModelFactory.java:22)
        at ro.isdc.wro.model.factory.WroModelFactoryDecorator.create(WroModelFactoryDecorator.java:27)
        at ro.isdc.wro.model.factory.ModelTransformerFactory.create(ModelTransformerFactory.java:55)
        at ro.isdc.wro.model.factory.ModelTransformerFactory.create(ModelTransformerFactory.java:27)
        at ro.isdc.wro.model.factory.DefaultWroModelFactoryDecorator$1.initialize(DefaultWroModelFactoryDecorator.java:60)
        at ro.isdc.wro.model.factory.DefaultWroModelFactoryDecorator$1.initialize(DefaultWroModelFactoryDecorator.java:50)
        at ro.isdc.wro.util.LazyInitializer.get(LazyInitializer.java:33)
        at ro.isdc.wro.model.factory.DefaultWroModelFactoryDecorator.create(DefaultWroModelFactoryDecorator.java:114)
        at ro.isdc.wro.model.factory.DefaultWroModelFactoryDecorator.create(DefaultWroModelFactoryDecorator.java:35)
        at ro.isdc.wro.model.group.processor.GroupsProcessor.process(GroupsProcessor.java:71)
        at ro.isdc.wro.cache.support.DefaultSynchronizedCacheStrategyDecorator.loadValue(DefaultSynchronizedCacheStrategyDecorator.java:101)
        at ro.isdc.wro.cache.support.DefaultSynchronizedCacheStrategyDecorator.loadValue(DefaultSynchronizedCacheStrategyDecorator.java:35)
        at ro.isdc.wro.cache.support.AbstractSynchronizedCacheStrategyDecorator.get(AbstractSynchronizedCacheStrategyDecorator.java:55)
        at ro.isdc.wro.manager.ResourceBundleProcessor.serveProcessedBundle(ResourceBundleProcessor.java:63)
        at ro.isdc.wro.manager.WroManager.process(WroManager.java:159)
        at ro.isdc.wro.http.WroFilter.processRequest(WroFilter.java:335)
        at ro.isdc.wro.http.WroFilter.doFilter(WroFilter.java:289)
        at org.fao.geonet.Geonetwork$1.run(Geonetwork.java:473)
        at java.lang.Thread.run(Thread.java:745)
2015-11-27 16:35:36,132 INFO  [geonetwork.wro4j] - Group highlight-json-xml contains:
        ...pps/geonetwork/data/formatter/xml_view/highlight-json-xml.js


Group highlightjs contains:
        ...a/webapps/geonetwork/data/formatter/xml_view/highlightjs.css


Group closure_deps contains:
        ...pps/geonetwork/data/formatter/xml_view/highlight-json-xml.js


2015-11-27 16:35:36,132 ERROR [geonetwork.wro4j] - Errors were encountered
2015-11-27 16:35:36,133 ERROR [geonetwork.wro4j] - error
java.lang.IllegalArgumentException: Parameter 'directory' is not a directory
        at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:460)
        at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:546)
        at org.apache.commons.io.FileUtils.iterateFiles(FileUtils.java:565)
        at org.fao.geonet.wro4j.GeonetWroModelFactory$ResourceDesc$1.iterator(GeonetWroModelFactory.java:707)
        at org.fao.geonet.wro4j.GeonetWroModelFactory.addCssGroupsByRequireDependencies(GeonetWroModelFactory.java:464)
        at org.fao.geonet.wro4j.GeonetWroModelFactory.loadGroupsUsingRequireDependencyManagement(GeonetWroModelFactory.java:395)
        at org.fao.geonet.wro4j.GeonetWroModelFactory.createModel(GeonetWroModelFactory.java:191)
        at org.fao.geonet.wro4j.GeonetWroModelFactory.createModel(GeonetWroModelFactory.java:163)
        at org.fao.geonet.wro4j.GeonetWroModelFactory.create(GeonetWroModelFactory.java:108)
        at org.fao.geonet.wro4j.GeonetWroModelFactory.create(GeonetWroModelFactory.java:65)
        at ro.isdc.wro.model.factory.WroModelFactoryDecorator.create(WroModelFactoryDecorator.java:27)
        at ro.isdc.wro.model.factory.FallbackAwareWroModelFactory.create(FallbackAwareWroModelFactory.java:48)
        at ro.isdc.wro.model.factory.FallbackAwareWroModelFactory.create(FallbackAwareWroModelFactory.java:22)
        at ro.isdc.wro.model.factory.WroModelFactoryDecorator.create(WroModelFactoryDecorator.java:27)
        at ro.isdc.wro.model.factory.ModelTransformerFactory.create(ModelTransformerFactory.java:55)
        at ro.isdc.wro.model.factory.ModelTransformerFactory.create(ModelTransformerFactory.java:27)
        at ro.isdc.wro.model.factory.DefaultWroModelFactoryDecorator$1.initialize(DefaultWroModelFactoryDecorator.java:60)
        at ro.isdc.wro.model.factory.DefaultWroModelFactoryDecorator$1.initialize(DefaultWroModelFactoryDecorator.java:50)
        at ro.isdc.wro.util.LazyInitializer.get(LazyInitializer.java:33)
        at ro.isdc.wro.model.factory.DefaultWroModelFactoryDecorator.create(DefaultWroModelFactoryDecorator.java:114)
        at ro.isdc.wro.model.factory.DefaultWroModelFactoryDecorator.create(DefaultWroModelFactoryDecorator.java:35)
        at ro.isdc.wro.model.group.processor.GroupsProcessor.process(GroupsProcessor.java:71)
        at ro.isdc.wro.cache.support.DefaultSynchronizedCacheStrategyDecorator.loadValue(DefaultSynchronizedCacheStrategyDecorator.java:101)
        at ro.isdc.wro.cache.support.DefaultSynchronizedCacheStrategyDecorator.loadValue(DefaultSynchronizedCacheStrategyDecorator.java:35)
        at ro.isdc.wro.cache.support.AbstractSynchronizedCacheStrategyDecorator.get(AbstractSynchronizedCacheStrategyDecorator.java:55)
        at ro.isdc.wro.manager.ResourceBundleProcessor.serveProcessedBundle(ResourceBundleProcessor.java:63)
        at ro.isdc.wro.manager.WroManager.process(WroManager.java:159)
        at ro.isdc.wro.http.WroFilter.processRequest(WroFilter.java:335)
        at ro.isdc.wro.http.WroFilter.doFilter(WroFilter.java:289)
        at org.fao.geonet.Geonetwork$1.run(Geonetwork.java:473)
        at java.lang.Thread.run(Thread.java:745)
2015-11-27 16:35:36,135 INFO  [geonetwork.wro4j] - StopWatch 'Create Wro Model using Geonetwork': running time (millis) = 504
-----------------------------------------
ms     %     Task name
-----------------------------------------
00504  100%  createModel

2015-11-27 16:35:36,155 ERROR [geonetwork.wro4j] - Error occurred during a wro4j request handling
ro.isdc.wro.WroRuntimeException: No such group available in the model: lib
        at ro.isdc.wro.model.group.processor.GroupsProcessor.process(GroupsProcessor.java:74)
        at ro.isdc.wro.cache.support.DefaultSynchronizedCacheStrategyDecorator.loadValue(DefaultSynchronizedCacheStrategyDecorator.java:101)
        at ro.isdc.wro.cache.support.DefaultSynchronizedCacheStrategyDecorator.loadValue(DefaultSynchronizedCacheStrategyDecorator.java:35)
        at ro.isdc.wro.cache.support.AbstractSynchronizedCacheStrategyDecorator.get(AbstractSynchronizedCacheStrategyDecorator.java:55)
        at ro.isdc.wro.manager.ResourceBundleProcessor.serveProcessedBundle(ResourceBundleProcessor.java:63)
        at ro.isdc.wro.manager.WroManager.process(WroManager.java:159)
        at ro.isdc.wro.http.WroFilter.processRequest(WroFilter.java:335)
        at ro.isdc.wro.http.WroFilter.doFilter(WroFilter.java:289)
        at org.fao.geonet.Geonetwork$1.run(Geonetwork.java:473)
        at java.lang.Thread.run(Thread.java:745)
2015-11-27 16:35:36,159 ERROR [geonetwork.wro4j] - Error occurred during a wro4j request handling
ro.isdc.wro.WroRuntimeException: No such group available in the model: nv.d3
        at ro.isdc.wro.model.group.processor.GroupsProcessor.process(GroupsProcessor.java:74)
        at ro.isdc.wro.cache.support.DefaultSynchronizedCacheStrategyDecorator.loadValue(DefaultSynchronizedCacheStrategyDecorator.java:101)
        at ro.isdc.wro.cache.support.DefaultSynchronizedCacheStrategyDecorator.loadValue(DefaultSynchronizedCacheStrategyDecorator.java:35)
        at ro.isdc.wro.cache.support.AbstractSynchronizedCacheStrategyDecorator.get(AbstractSynchronizedCacheStrategyDecorator.java:55)
        at ro.isdc.wro.manager.ResourceBundleProcessor.serveProcessedBundle(ResourceBundleProcessor.java:63)
        at ro.isdc.wro.manager.WroManager.process(WroManager.java:159)
        at ro.isdc.wro.http.WroFilter.processRequest(WroFilter.java:335)
        at ro.isdc.wro.http.WroFilter.doFilter(WroFilter.java:289)
        at org.fao.geonet.Geonetwork$1.run(Geonetwork.java:473)
        at java.lang.Thread.run(Thread.java:745)
2015-11-27 16:35:36,162 ERROR [geonetwork.wro4j] - Error occurred during a wro4j request handling
ro.isdc.wro.WroRuntimeException: No such group available in the model: gn_search_default

and lots more. Nore that if ${geonetwork.datadir}/data/formatter is empty, it is autofilled with the content of ${webappdir}/WEB-INF/data/data/formatter/* so that's probably somewhat related.

Anyway, i shouldnt have upgraded :)

@pmauduit pmauduit added a commit to pmauduit/core-geonetwork that referenced this issue Nov 29, 2015
@pmauduit pmauduit Fixing h2 management for wro4j
- DB_CLOSE_DELAY=-1 is a bad idea in our case because it is meant for
  in-memory databases, which is not our case here ; and it seems to play
  a negative role when dealing with redeploys
- Firing up a SHUTDOWN sql statement before actually closing the db when
  destroying
- Ensure the DiskbackedCache is actually destroyed, so that the DB is
  shut down and the lock is released across redeploys

Related issues:
georchestra/georchestra#1120
geonetwork#1158
a1a8c6e
@landryb
Member
landryb commented Nov 30, 2015

Issue in my last comment fixed, local git submodules messup within geonetwork submodule within georchestra.. i had no bootstrap submodules, and that broke wro4j at runtime in... mysterious ways..

@pmauduit pmauduit added a commit to pmauduit/core-geonetwork that referenced this issue Dec 8, 2015
@pmauduit @pmauduit pmauduit + pmauduit Fixing h2 management for wro4j
- DB_CLOSE_DELAY=-1 is a bad idea in our case because it is meant for
  in-memory databases, which is not our case here ; and it seems to play
  a negative role when dealing with redeploys
- Firing up a SHUTDOWN sql statement before actually closing the db when
  destroying
- Ensure the DiskbackedCache is actually destroyed, so that the DB is
  shut down and the lock is released across redeploys

Related issues:
georchestra/georchestra#1120
geonetwork#1158
04fdffc
@pmauduit
Member
pmauduit commented Dec 8, 2015

Needs a merge / rebase now that some PRs related to this issue have been merged

@landryb
Member
landryb commented Dec 8, 2015

Fwiw here's the branch i'm using on my pre-production instances, with recent commits to 3.0.x merged and some other pending cherrypicked on top : https://github.com/landryb/core-geonetwork/tree/integration-georchestra-gn3-3.0.x

@landryb
Member
landryb commented Dec 10, 2015

While working on this, i started tinkering with spring overrides to figure a way to properly set all geonetwork datadir subdirs from our georchestra datadir (cf https://github.com/geonetwork/core-geonetwork/blob/develop/core/src/main/java/org/fao/geonet/kernel/GeonetworkDataDirectory.java#L267)

So far, here's what i tried:

  • setting GeonetworkDataDirectory.htmlCacheDir=/tmp/htmlcache2 directly in geonetwork-security-overrides.properties -> ignored
  • overriding the GeonetworkDataDirectory bean init (originally in config-spring-geonetwork.xml) in a new override file containing:
  <bean id="GeonetworkDataDirectory" class="org.fao.geonet.kernel.GeonetworkDataDirectory">
    <property name="htmlCacheDir" value="/tmp/htmlcache1" />
  </bean>

Almost there...

Caused by: org.springframework.beans.ConversionNotSupportedException: Failed to convert property value of type 'java.lang.String' to required type 'java.nio.file.Path' for property 'htmlCacheDir'
@landryb
Member
landryb commented Dec 10, 2015

Even after working around this cast issue, it seems my properties arent properly used. Will dig more.

@landryb
Member
landryb commented Dec 10, 2015

https://github.com/landryb/core-geonetwork/tree/feature/datadir-bean-property adds the missing glue in geonetwork to be able to specify all paths as properties to the bean, using something like:

# in config-overrides-georchestra.xml
<import file="${env:georchestra.datadir}/geonetwork/config/config-datadir-georchestra.xml" />
# config-datadir-georchestra.xml
<beans>
  <context:property-placeholder location="file:${georchestra.datadir}/geonetwork/geonetwork.properties" file-encoding="UTF-8" ignore-unresolvable="true" order="0"/>

  <bean id="GeonetworkDataDirectory" class="org.fao.geonet.kernel.GeonetworkDataDirectory">
    <property name="systemDataDir" ref="GNSystemDataDir"/>
    <property name="schemaPluginsDir" ref="GNSchemaPluginsDir"/>
  </bean>
  <bean id="GNSystemDataDir" class="java.nio.file.Paths" factory-method="get">
    <constructor-arg index="0" value="${geonetwork.dir}"/>
    <constructor-arg index="1"><array /></constructor-arg>
  </bean>
  <bean id="GNSchemaPluginsDir" class="java.nio.file.Paths" factory-method="get">
    <constructor-arg index="0" value="${geonetwork.schema.dir}"/>
    <constructor-arg index="1"><array /></constructor-arg>
  </bean>
<beans>
# geonetwork.properties
geonetwork.dir=/data/webapps/geonetwork
geonetwork.schema.dir=/data/webapps/geonetwork/config/schema_plugins

Upstream PR at geonetwork/core-geonetwork#1347

@landryb
Member
landryb commented Dec 15, 2015

Another 'nice to have but can wait later' tweak would be to allow getting log4j.xml from ${georchestra.datadir} like other modules, possibly pushing this upstream. Dunno if that's possible via overrides, but GN3 has runtime-switching of loglevels (code partly at https://github.com/geonetwork/core-geonetwork/blob/develop/services/src/main/java/org/fao/geonet/services/config/LogUtils.java) like geoserver does, and this between all the WEB-INF/classes/log4j*xml files. Could be nice to keep this, only pointing the dir to ${georchestra.datadir}/geonetwork/log4j, but i'm not sure we can reuse the plumbing in georchestra-commons for this without invasive (and unmergeable upstream) changes.

@pmauduit
Member

Still TODO (Nice to have) for georchestra integration:

  • Log4j configuration into georchestra datadir (#1154)
  • Spatial indexes in postgis (#1139)
@fvanderbiest
Member

Also, we have to check everything works as expected with mapfishapp, both for GN2 and GN3.

$ g grep geonetwork
src/deb/resources/etc/georchestra/mapfishapp/js/GEOR_custom.js:     * Defaults to "/geonetwork"
src/deb/resources/etc/georchestra/mapfishapp/js/GEOR_custom.js:    GEONETWORK_BASE_URL: "http://geobretagne.fr/geonetwork",
src/deb/resources/etc/georchestra/mapfishapp/js/GEOR_custom.js:        ['http://sdi.georchestra.org/geonetwork/srv/fre/csw', 'le catalogue geOrchestra démo'],
src/deb/resources/etc/georchestra/mapfishapp/js/GEOR_custom.js:        ['http://geobretagne.fr/geonetwork/srv/fre/csw', 'le catalogue GeoBretagne'],
src/deb/resources/etc/georchestra/mapfishapp/js/GEOR_custom.js:        ['http://ids.pigma.org/geonetwork/srv/fre/csw', 'le catalogue PIGMA'],
src/deb/resources/etc/georchestra/mapfishapp/js/GEOR_custom.js:        ['/geonetwork/srv/fre/csw', 'le catalogue local'],
src/deb/resources/etc/georchestra/mapfishapp/js/GEOR_custom.js:        ['http://sandre.eaufrance.fr/geonetwork_CSW/srv/fre/csw', 'le catalogue du Sandre'],
src/deb/resources/etc/georchestra/mapfishapp/js/GEOR_custom.js:        ['http://geocatalog.webservice-energy.org/geonetwork/srv/fre/csw', 'le catalogue de webservice-energy'],
src/deb/resources/etc/georchestra/mapfishapp/js/GEOR_custom.js:        ['http://www.ifremer.fr/geonetwork/srv/fre/csw', "le catalogue de l'Ifremer"]
src/deb/resources/etc/georchestra/mapfishapp/js/GEOR_custom.js:    DEFAULT_CSW_URL: 'http://sdi.georchestra.org/geonetwork/srv/fre/csw',
src/deb/resources/etc/georchestra/mapfishapp/js/GEOR_custom.js:     * Key (as the one in the response from /geonetwork/srv/fre/xml.thesaurus.getList) 
src/main/webapp/app/css/main.css: max-width: 180px; /* to cope with larger images (geonetwork has 180px thumbnails)*/
src/main/webapp/app/js/GEOR_config.js:         * Defaults to "/geonetwork"
src/main/webapp/app/js/GEOR_config.js:            "/geonetwork"),
src/main/webapp/app/js/GEOR_config.js:            ['http://sdi.georchestra.org/geonetwork/srv/fre/csw', 'le catalogue geOrchestra démo'],
src/main/webapp/app/js/GEOR_config.js:            ['/geonetwork/srv/fre/csw', 'le catalogue local']
src/main/webapp/app/js/GEOR_config.js:            'http://sdi.georchestra.org/geonetwork/srv/fre/csw'),
src/main/webapp/app/js/GEOR_config.js:         * Key (as the one in the response from /geonetwork/srv/fre/xml.thesaurus.getList)
src/main/webapp/app/js/GEOR_cswbrowser.js:     * {String} eg: /geonetwork/srv/fre
src/main/webapp/app/js/GEOR_cswquerier.js:                // this part is 100% geonetwork specific:
src/main/webapp/app/js/GEOR_cswquerier.js:                    // this part is 100% geonetwork specific:
src/main/webapp/app/js/GEOR_cswquerier.js:            // improve relevance of results: (might not be relevant with other csw servers than geonetwork)
@landryb
Member
landryb commented Dec 22, 2015

The editing toolbar is properly positioned after georchestra/geonetwork@172c318

@pmauduit
Member

@landryb And I wonder if we should not display: none on the bottom bar ... It does not provide anything useful actually (a link to the GN github repository)

@landryb
Member
landryb commented Dec 22, 2015

I think it's somewhat nice to keep it, since ppl might want to customize it, there's no harm to advertise we use geonetwork 3.0.x, and by default it also links to the rss feed....

Would be nice to fix the internal viewer sizing too :)

@fvanderbiest
Member

Closing this one, new issues should be created for remaining bugs or new ones.

@pmauduit pmauduit added a commit to pmauduit/geonetwork that referenced this issue Jan 13, 2017
@pmauduit @pmauduit pmauduit + pmauduit security-overrides in its own file + sourcing it in the xml
See
georchestra/georchestra#1120 (comment)
for the motivation behind this commit.

Tests: Runtime
da6daf8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment