Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improvements to allow future deployment of multiple instances #6990

Closed
wants to merge 6 commits into from
Closed

Improvements to allow future deployment of multiple instances #6990

wants to merge 6 commits into from

Conversation

fxprunayre
Copy link
Member

@fxprunayre fxprunayre commented Apr 14, 2023 •
edited
Loading

HTML cache for formatter and WRO4J

This allows to have multiple instances starting up but having their own separate HTML cache. If not, H2 database are locked.

Caused by: org.h2.mvstore.MVStoreException: The file is locked: /.../WEB-INF/data/data/resources/htmlcache/wro4j-cache.mv.db [2.1.212/7]
	at org.h2.mvstore.DataUtils.newMVStoreException(DataUtils.java:1004) ~[h2-2.1.212.jar:2.1.212]

geonetwork.htmlcache.dir can be used to cusomize the location. Another workaround is to use LRU cache instead

cacheStrategy=disk-memory

replaced by

cacheStrategy=lru

to have in memory cache (but you can't benefit from the pre-built cache).

Schema initialization

On startup, GeoNetwork publish each schema plugin XSDs to /data/resources/xml/schemas folder. If more that one
instance starts at the same time, they can fails to copy the XSD. Adding a new data directory variables for the schemas -Dgeonetwork.schemapublication.dir=.

Admin / Site info

Added all config keys to make easier configuration check on each instance:

image

Harvester

Add configuration to only run scheduled harvesting tasks in one instance.
See #7217

Future improvements

  • Editing session in parallel on same record - locking system to not allow multiple editing session

@fxprunayre fxprunayre added this to the 4.2.4 milestone Apr 14, 2023
@sonarcloud
Copy link

sonarcloud bot commented Apr 14, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

@fxprunayre fxprunayre changed the title WRO4J / Move cache to HTML cache folder. Multiple instances exercise Apr 14, 2023
@fxprunayre fxprunayre modified the milestones: 4.2.4, 4.2.5 Apr 24, 2023
@fxprunayre
WRO4J / Move cache to HTML cache folder.
81c3257 
This allows to have multiple instances starting up but having their own separate HTML cache. If not H2 database are locked.
@fxprunayre @joachimnielandt
Data directory / Add parameter for XML schemas.
1300e5b 
On startup, GeoNetwork publish each schema plugin XSDs to /data/resources/xml/schemas folder. If more that one
instance starts at the same time, they can fails to copy the XSD. Adding a new data directory variables for the schemas -Dgeonetwork.schemapublication.dir=.

Co-authored-by: Joachim Nielandt <joachim.nielandt@vlaanderen.be>
@fxprunayre fxprunayre force-pushed the 424-multipleinstances branch 2 times, most recently from abce2ca to 82063e0 Compare June 26, 2023 07:16
@joachimnielandt @fxprunayre
API / Site info / Add all data directory info
f078d30 
* Sort by insertion order
* Showing data directory html cache directory setting in admin page

Co-authored-by: Joachim Nielandt <joachim.nielandt@vlaanderen.be>
@sonarcloud
Copy link

sonarcloud bot commented Jun 27, 2023

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 4 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

idea Catch issues before they fail your Quality Gate with our IDE extension sonarlint SonarLint

@fxprunayre fxprunayre modified the milestones: 4.2.5, 4.2.6, 4.4.0 Jul 5, 2023
@fxprunayre fxprunayre mentioned this pull request Jul 12, 2023
Closed
CMath04 pushed a commit to Informatievlaanderen/metadata-geonetwork that referenced this pull request Aug 2, 2023
@joachimnielandt
Merged PR 24976: 1.0.2 - High Availability fixes
9e2a466 
follow-up of: geonetwork#6990

Introduce docker-scalable mode for geonetwork for easy testing and separate out the html cache for each container.

Related work items: #178184, #178190
@fxprunayre fxprunayre marked this pull request as ready for review August 23, 2023 05:33
josegar74
josegar74 reviewed Aug 28, 2023
View reviewed changes
core/src/main/java/org/fao/geonet/resources/ResourceFilter.java

private String extensionToMediaType(String ext) {
final String contentType;
if(Sets.newHashSet("xml", "xsd", "sch", "dtd").contains(ext)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe move Sets.newHashSet("xml", "xsd", "sch", "dtd") to an instance variable to avoid creating the set on every request?

web-ui/src/main/resources/catalog/locales/fr-admin.json Show resolved Hide resolved
web-ui/src/main/resources/catalog/locales/nl-admin.json Show resolved Hide resolved
@josegar74
Copy link
Member

@fxprunayre this needs to be updated to web/src/main/webapp/WEB-INF/data/data/resources/schemapublication:

core-geonetwork/.gitignore

Line 71 in 71d40a1

web/src/main/webapp/WEB-INF/data/data/resources/xml

josegar74
josegar74 reviewed Aug 28, 2023
View reviewed changes
Copy link
Member

@josegar74 josegar74 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested with a single instance and looks working fine.

Before approving the pull request, I think should be provided some documentation about how to configure with multiple instances and test that scenario.

@fxprunayre
Copy link
Member Author

Before approving the pull request, I think should be provided some documentation about how to configure with multiple instances and test that scenario.

We are testing it further this week with Vlaanderen. We are planning to provide more info how to do this with docker which will explains how to configure stuff. But first want to be sure that one instance setup is all fine. And this PR should be fine.

@fxprunayre fxprunayre changed the title Multiple instances exercise Improvements to allow future deployment of multiple instances Aug 28, 2023
@sonarcloud
Copy link

sonarcloud bot commented Aug 28, 2023

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 3 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

warning The version of Java (11.0.20) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more here

idea Catch issues before they fail your Quality Gate with our IDE extension sonarlint SonarLint

This was referenced Sep 13, 2023
Merged
Merged
@fxprunayre
Copy link
Member Author

In #7337

@fxprunayre fxprunayre closed this Sep 13, 2023
@fxprunayre fxprunayre deleted the 424-multipleinstances branch September 19, 2023 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@josegar74 josegar74 josegar74 left review comments

Assignees
No one assigned
Labels
changelog
Projects
None yet
Milestone
4.4.0
Development

Successfully merging this pull request may close these issues.
3 participants
@fxprunayre @josegar74 @joachimnielandt