Skip to content

Java 7 NIO File

Jump to bottom Edit New page
juanluisrp edited this page Nov 17, 2014 · 4 revisions
Date 2014-11-13 Contacts Jesse Eichar
Status Voting Release 3.0
Ticket # https://github.com/geonetwork/core-geonetwork/issues/676
Source code https://github.com/jesseeichar/core-geonetwork/tree/replace_file_with_path

Overview

This change replaces the use of java.io.File with java.nio.file.Path. The Java 7 API is much more flexible and powerful than the java.io.File API. For example, the paths do not have to on the local file system but can be in zip files, in memory, in database, etc...

Currently running the full build between 30 minutes and an hour. Largely because the integration tests require a great deal of time during setup and teardown of each test.

As a result geonetwork trunk has had failing tests for the last 6 months or more.

Using the Path API instead of java.io.File allows the tests to use an in memory filesystem instead of one on disk. This has two effects.

  1. It greatly decreases the time to do a build. After this change the building the core module only takes: 3 minutes instead of `> 10 minutes' required by the current build. A full build takes 5-10 minutes.
  2. It reduces the wear of hard drives that need to perform tens or hundreds of thousands of copies during a Geonetwork build.

This is the primary drive for this change however there are secondary benefits of this change. For example:

  • Alternate FileSystems can be used to store the Geonetwork DataDirectory. Some ideas are:
  • In memory
  • In a database
  • In the cloud
  • The new API has a watch API to get notifications when a file changes, instead of having to perform datestamp comparisons Geonetwork could register a watcher. This would be very useful for the Localfilesystem harvester
  • The new API has the ability to handle atomic directory and file operations.

Technical Details:

This change makes the following changes:

  • Replaces all uses of java.io.File with java.nio.file.Path.
  • Replaces most strings that represent paths with Path.
  • The XmlResolver takes the system id (as a URI) and converts it to a path and resolves the href against that base path to find the final path. Since a path is created from a URI the path is not necessarily a path to a file on the file system. In this way the files can be on any Java 7 NIO file filesystem
  • In addition Xml.load uses a similar strategy when loading xml files. It has to use a ThreadLocal variable to do this in this case however because the API is slightly different and the path of the source is not provided. However the threadlocal is used to set the base file and all imports are imported relative to that file. This is primarily required for XSD files.
  • Validation uses the XmlResolver
  • The AbstractCoreIntegrationTest now loads the datadirectory into a Memory FS and keeps one copy as a clean copy. Each test that is created (after the first) take a copy of the clean in-memory FS and uses that new copy for running its tests. This means that for each module the data is loaded from the file system only a single time and no writes are done.

Proposal Type:

  • Type: Improvement
  • Module: All

Voting History

  • Vote Proposed: TBA

Participants

  • All
Add a custom footer
Add a custom sidebar
Clone this wiki locally