Support indexing the ISO files from the IOOS Catalog #574
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #574 +/- ##
==========================================
+ Coverage 55.81% 56.41% +0.59%
==========================================
Files 28 28
Lines 6355 6401 +46
Branches 1349 1357 +8
==========================================
+ Hits 3547 3611 +64
+ Misses 2422 2401 -21
- Partials 386 389 +3
Continue to review full report at Codecov.
|
kwilcox
changed the title
|
I'm running the tests against sqlite and postgres to make sure the VACUUM fix is actually fixed... |
Carry through the actual exception when trying to insert records
This turns ``` <ows:ExceptionText>Harvest failed: record parsing failed: local variable 'md' referenced before assignment</ows:ExceptionText> ``` into ``` <ows:ExceptionText>Harvest failed: record parsing failed: 404 Client Error: Not Found for url: http://maps.cera.govt.nz/arcgis/services/CERA/CERA_TechClasses_WGS84/MapServer/WFSServer?service=WFS&request=GetCapabilities&version=2.0.0</ows:ExceptionText> ```
…ython#566 Return the loades and exported records from the admin functions Export records in functional tests if an "export" directory exists I spent a good amount of time trying to integrate this into the testing suite more cleanrly (not in the _initialize_database function) but came up with no solution that didn't require a large refator of the testing code.
kwilcox
changed the title
|
@tomkralidis ready for review |
tomkralidis
reviewed
Aug 27, 2018
| msg = 'Cannot commit to repository' | ||
| LOGGER.exception(msg) | ||
| raise RuntimeError(msg) | ||
| raise |
There was a problem hiding this comment.
Will (only) raise emit msg?
There was a problem hiding this comment.
Yup, this just re-raises the same exception that was caught in the except block. This was used so I could catch DB errors in the admin.load_records function and return a specific error rather than a generic message/exception.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
I was attempting to setup a
pycswserver for IOOS and ran across a few hiccups that I fixed in this PR. I also fixed an outstanding issue (#561) that I ran into when running the functional tests.Related Issue / Discussion
In general: it was not easy to debug loading a very large folder of ISO files into
pycsw. The loops would error out if a file couldn't be parsed and the error messages were not specific enough to debug which files were cause the issues. In some cases the error messages were much to verbose because they contained the entire SQL INSERT statement. The logging when loading records is much more user friendly now. While making the logging more friendly I came across a few other hiccups that I fixed along the way.Additional Information
Contributions and Licensing
(as per https://github.com/geopython/pycsw/blob/master/CONTRIBUTING.rst#contributions-and-licensing)