Replace fgdc2iso service with CLI invocation #2917
Comments
mogul
changed the title
|
Can someone add pointers needed for getting the JRE and Saxon-Java/HE .debs included in the Ansible playbook to the sketch above? |
Currently we're not running harvesters and looking to save some memory quota. Also, we're planning to [replace the fgdc2iso service][1]. [1]: GSA/data.gov#2917
|
I was able to successfully add java to the current catalog docker image via apk; add the necessary saxon he jar file, and run a command to transform the current FGDC/CSDGM example in catalog-fgdc2iso into ISO! |
|
I added a test to geodatagov, which is failing. The error has to do with the encoding of the xml file. I based the new transformation return on the old service transformation return. Something must be happening with the way the server returns/encodes the result. |
|
By removing the Also, removed all FGDC deployments on cloud.gov: |
|
CloudFoundry doesn't like the java installation via apt... |
|
You might still need to include the |
😬 |
Looks like someone has summarized concisely what needs to be done at SO. |
|
I did see that, although my experience was not the same. Possibly because I'm installing the latest known version to work with saxon transformation (11.x). I was able to use |
|
Tried both installing java 8 (and other versions, same error) and changing soft links (different error). This is definitely an issue with the install. Going to try to install java via apt in ubuntu image and see if the saxon file can be run... |
|
Status: adding the |
|
We tested installing |
|
I believe I've found a way we can precede the installation of the package with the
I suspect we can create a "profile" .deb file that marks |
|
(If |
|
I take it back; it looks like we can't use a "profile" package to do this because of when
The language here suggests that we can't rely on However, the answer is still pretty simple... The package we create should just include the (I looked up how |
|
Here's a little present for you, @jbrown-xentity... It should probably get broken out into a top-level repository since it can be useful to people using CF and the apt-buildpack generally, but I'll leave that as an exercise for you. 😜 [Edited to add: Attention Google searchers, this |
|
You can install |
Progress update:We can get the After this, However, that's not enough because when you run the actual saxon transform it flames out: [EXPAND] java barfs on broken links in installThis is because the OpenJDK installation ships with a bunch of symlinks that point to /etc. The symlinks are broken because the apt-buildpack doesn't run as root and doesn't install the expected stuff at /etc... They are installed at /home/vcap/deps/0/apt/etc instead. So we need to fix up those links. This one-liner works: That fixes the 25 broken links to point to It turns out there is still one broken link after that fix: To properly start an SSL client connection, To verify this was the problem I tried copying the SUCCESS! I did a little additional work and now the test command works after SSH'ing in and just sourcing .profile. I pushed this working test to the Now the question is: Are we content with shipping our own Here's a log of those commands in action: [EXPAND]: Full log of generating cacertsThe question I still have is: Is that the right thing to do? By including the Still a mystery: Why doesn't the |
|
apt-test.zip |
User Story
In order to remove moving parts and reduce the compliance overhead of operating data.gov, the data.gov team wants to convert geometadata formats during harvesting using a CLI call instead of calling a separate web service.
Acceptance Criteria
ckanext-geodatagovplugin repositoryWHEN I run the tests
THEN I see a test for conversion of FGDC/RSE to ISO 19115-2 listed
AND the test passes
datagov-deployrepository's issue tracker for issues labeledcomponent/fgdc2isoAND I search the
datagov-deployrepository's issue tracker for issues mentioning fgdc2iso in the titleTHEN I find only closed issues
catalog-fgdc2isorepository on GitHubTHEN I see the repository is archived, with a note that it has been replaced
AND I have run
cf t -o gsa-datagovWHEN I run
cf t -s development ; cf apps | grep fgdc2isoAND I run
cf t -s staging ; cf apps | grep fgdc2isoAND I run
cf t -s prod ; cf apps | grep fgdc2isoTHEN I do not see any results.
Background
Here's the results of the research story that prompted this issue.
Security Considerations (required)
This change will add the JRE runtime to the catalog.data.gov application in order to allow for calling a .jar from the CLI. It will also remove the need for an entire webservice, catalog-fgdc2iso, and communications with that service. The net result will be a much simpler system with fewer potentially-exposed surfaces to protect
Include a test for XXE handling if applicable (#3246).
Sketch
Here's the code that calls fgdc2iso now, and the Jython code that actually does the transform now.
apt install.java net.sf.saxon.Transform -s:[source] -xsl:[stylesheet] -o:[output]apt.yamlfile to install Saxon-HE/Java from a PPA.The text was updated successfully, but these errors were encountered: