API / CSV export / Add support for custom export.

[fxprunayre]

Current CSV export is based on XSL transformations and can be hard to use when user is interested in element with multiple values (eg. online source, contacts).

Default CSV export remains the same but add the possibility to customize the export with 2 additional parameters to create custom export:

• loopElementXpath: element to loop on eg.

  • use . for the metadata
  • .//gmd:CI_ResponsibleParty for all contacts in ISO19139,
  • .//gmd:transferOptions/*/gmd:onLine/* for all online resources in ISO19139.

• propertiesXpath: columns to extract eg.

  • gmd:identificationInfo/*/gmd:citation/*/gmd:title//text() for the title.

Also add parameters for defining separators:

• sep for column separator
• internalSep when multiple values are stored in a field

User can then build custom reports from the API:

eg. Export Contact with role,org,email
http://localhost:8080/srv/api/records/csv?bucket=s101&loopElementXpath=.//gmd:CI_ResponsibleParty&propertiesXpath=gmd:role/*/@codeListValue&propertiesXpath=gmd:organisationName/*/text()&propertiesXpath=.//gmd:electronicMailAddress/*/text()

eg. Export Online source with protocol,url,name,desc http://localhost:8080/srv/api/records/csv?bucket=s101&loopElementXpath=.//gmd:transferOptions/*/gmd:onLine/*&propertiesXpath=gmd:protocol/*/text()&propertiesXpath=gmd:linkage/*/text()&propertiesXpath=gmd:name/*/text()&propertiesXpath=gmd:description/*/text()

eg. Export Metadata with title,alternateTitle,status,maintenanceFreq,... http://localhost:8080/srv/api/records/csv?bucket=s101&loopElementXpath=.&propertiesXpath=gmd:identificationInfo/*/gmd:citation/*/gmd:title//text()&propertiesXpath=gmd:identificationInfo/*/gmd:citation/*/gmd:alternateTitle//text()&propertiesXpath=gmd:identificationInfo/*/gmd:status/*/@codeListValue&propertiesXpath=gmd:identificationInfo/*//gmd:maintenanceAndUpdateFrequency/*/@codeListValue&propertiesXpath=gmd:identificationInfo/*//gmd:otherConstraints//text()&propertiesXpath=gmd:identificationInfo/*/gmd:topicCategory//text()&propertiesXpath=gmd:identificationInfo/*/gmd:language/*/@codeListValue&propertiesXpath=gmd:identificationInfo/*/gmd:graphicOverview/*/gmd:fileName/*/text()

Functions from XPath v1 can also be used eg. count http://localhost:8080/srv/api/records/csv?bucket=s101&loopElementXpath=.&propertiesXpath=count(gmd:identificationInfo/*/gmd:descriptiveKeywords)

For all exports 2 columns are added first:

• UUID
• permalink

When using XPath, it is recommended to export records in same schema (or at least same base schema). If not, then XPath error messages are returned in cells unless XPath provided do not require namespaces. Users have to configure a proper selection to avoid mixing schema.

[sonarcloud]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

0.0% Coverage
0.0% Duplication

[bernhardreiter]

The following patch fixes the code smell found by sonarlint (and improves the comment http -> 'https').

diff --git a/services/src/main/java/org/fao/geonet/api/records/CatalogApi.java b/services/src/main/java/org/fao/geonet/api/records/CatalogApi.java
index e935046aad..4034cae6a8 100644
--- a/services/src/main/java/org/fao/geonet/api/records/CatalogApi.java
+++ b/services/src/main/java/org/fao/geonet/api/records/CatalogApi.java
@@ -125,6 +125,7 @@ public class CatalogApi {
             .add("resourceTitleObject.default") // TODOES multilingual
             .add("resourceAbstractObject.default").build();
     }
+    private static final String AMP_ENT ="&";
 
     @Autowired
     DefaultLanguage defaultLanguage;
@@ -780,14 +781,14 @@ public class CatalogApi {
         String previousPage = canonicalURL + "?" + paramsAsString(allRequestParams) + "&from=" + prevFrom + "&to=" + prevTo;
         String nextPage = canonicalURL + "?" + paramsAsString(allRequestParams) + "&from=" + nextFrom + "&to=" + nextTo;
 
-        // Hydra Paging information (see also: http://www.hydra-cg.com/spec/latest/core/)
-        String hydraPagedCollection = "<hydra:PagedCollection xmlns:hydra=\"http://www.w3.org/ns/hydra/core#\" rdf:about=\"" + currentPage.replace("&", "&amp;") + "\">\n" +
+        // Hydra Paging information (see also: https://www.hydra-cg.com/spec/latest/core/)
+        String hydraPagedCollection = "<hydra:PagedCollection xmlns:hydra=\"http://www.w3.org/ns/hydra/core#\" rdf:about=\"" + currentPage.replace("&", AMP_ENT) + "\">\n" +
             "<rdf:type rdf:resource=\"hydra:PartialCollectionView\"/>" +
-            "<hydra:lastPage>" + lastPage.replace("&", "&amp;") + "</hydra:lastPage>\n" +
+            "<hydra:lastPage>" + lastPage.replace("&", AMP_ENT) + "</hydra:lastPage>\n" +
             "<hydra:totalItems rdf:datatype=\"http://www.w3.org/2001/XMLSchema#integer\">" + numberMatched + "</hydra:totalItems>\n" +
-            ((prevFrom <= prevTo && prevFrom < from && prevTo < to) ? "<hydra:previousPage>" + previousPage.replace("&", "&amp;") + "</hydra:previousPage>\n" : "") +
-            ((nextFrom <= nextTo && from < nextFrom && to < nextTo) ? "<hydra:nextPage>" + nextPage.replace("&", "&amp;") + "</hydra:nextPage>\n" : "") +
-            "<hydra:firstPage>" + firstPage.replace("&", "&amp;") + "</hydra:firstPage>\n" +
+            ((prevFrom <= prevTo && prevFrom < from && prevTo < to) ? "<hydra:previousPage>" + previousPage.replace("&", AMP_ENT) + "</hydra:previousPage>\n" : "") +
+            ((nextFrom <= nextTo && from < nextFrom && to < nextTo) ? "<hydra:nextPage>" + nextPage.replace("&", AMP_ENT) + "</hydra:nextPage>\n" : "") +
+            "<hydra:firstPage>" + firstPage.replace("&", AMP_ENT) + "</hydra:firstPage>\n" +
             "<hydra:itemsPerPage rdf:datatype=\"http://www.w3.org/2001/XMLSchema#integer\">" + hitsPerPage + "</hydra:itemsPerPage>\n" +
             "</hydra:PagedCollection>";
         // Construct the RDF output

[sonarcloud]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

0.0% Coverage
0.0% Duplication

The version of Java (11.0.20) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more here

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

[benoitregamey]

Tested it out, works fine. Very useful for many users !