You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Set up Elasticsearch & openly available kibana with complete Loc-instances-Bibframe dataset #23
changed the title [-]Set up Elasticsearch & openly available kibana with complete Bibframe dataset[/-][+]Set up Elasticsearch & openly available kibana with complete Loc-instances-Bibframe dataset[/+]on Nov 5, 2018
While the solution provided in a) hbz/lobid-resources#851 is not proper enough but b) #2 not fast enough we go again with a) using java-jsonld, facing some issues which must be resolved, see hbz/lobid-resources#940.
@dr0i is now able to convert the LOC data with the Java JSON-LD library and the output looks much like what we generate in the workshop. Here is the current status with the same example that we use in the workshop:
Did that. Also indexed a 100k test resources , see curl "http://weywot5.hbz-nrw.de:9200/loc_works/_search?q=*" |json_pp. ETL time was 5 minutes so it takes roughly the same time as the ETL of lobid-resources.
Just took a look at the data. Here is what I noticed.
Some things are missing in the context as MADS classes are not compacted: http://www.loc.gov/mads/rdf/v1#ComplexSubject, http://www.loc.gov/mads/rdf/v1#GenreForm, http://www.loc.gov/mads/rdf/v1#Temporal, http://www.loc.gov/mads/rdf/v1#Geographic etc. also the MADSproperty http://www.loc.gov/mads/rdf/v1#isMemberOfMADSScheme. Also some bflc properties: http://id.loc.gov/ontologies/bflc/consolidates, http://id.loc.gov/ontologies/bflc/relatorMatchKey, http://id.loc.gov/ontologies/bflc/applicableInstitution where the first two are also missing in the workshop context (see Some property/type URIs are not compacted #32).
Why not use the context at https://github.com/hbz/swib18-workshop/blob/master/data/context.json? (It would be easier for me to work with one context anyway.)
Kibana is quite helpful for finding other errors. When I got to "Management" -> "Index Patterns" -> select "loc_works" and then filter by http:// I get all fields with a "http" in them. There are those already noted in 1.) and some more that have in common that they have "@language": null in the context at https://github.com/hbz/lobid-resources/blob/523-addLanguageTagSupport/web/conf/context-loc.jsonld
I guess, nearly all errors will be fixed with using the workshop context. The rest will be fixed with #32.
BTW, a lessons learned for Kibana: At "Management" -> "index patterns" -> "loc_works" I had to refresh the field list. Otherwise the fields from the previous index would be shown there and also in the field autosuggests for creating a visualization.
@acka47 I indexed loc-works-bibframe with the workshop's json-context. You have to set the index-alias to use it for kibana. 3k more documents are missing now, you will get tomorrow morning the lobid-admin mail complaining about [MapperParsingException[object mapping for [language] tried to parse field [null] and some more for subject etc. and the known adminMetadata.changeDate.
Activity
[-]Set up Elasticsearch & openly available kibana with complete Bibframe dataset[/-][+]Set up Elasticsearch & openly available kibana with complete Loc-instances-Bibframe dataset[/+]dr0i commentedon Nov 5, 2018
While the solution provided in a) hbz/lobid-resources#851 is not proper enough but b) #2 not fast enough we go again with a) using java-jsonld, facing some issues which must be resolved, see hbz/lobid-resources#940.
acka47 commentedon Nov 9, 2018
@dr0i is now able to convert the LOC data with the Java JSON-LD library and the output looks much like what we generate in the workshop. Here is the current status with the same example that we use in the workshop:
The current differences to the JSON-LD from the workshop (at least those I have found) are:
http://id.loc.gov/ontologies/bflc/EncodingLevelis mapped toEncodedLevelin the context (instead ofEncodingLevel), see https://github.com/hbz/lobid-resources/blob/523-addLanguageTagSupport/web/conf/context-loc.jsonld#L65titleSortKeyhas an array ("@container": "@set"in the context)Improve loc context; add runner
dr0i commentedon Nov 12, 2018
Did that. Also indexed a 100k test resources , see
curl "http://weywot5.hbz-nrw.de:9200/loc_works/_search?q=*" |json_pp. ETL time was 5 minutes so it takes roughly the same time as the ETL of lobid-resources.acka47 commentedon Nov 12, 2018
Thanks for indexing. Will soon take a look.
\o/ This is good to hear.
acka47 commentedon Nov 13, 2018
Just took a look at the data. Here is what I noticed.
http://www.loc.gov/mads/rdf/v1#ComplexSubject,http://www.loc.gov/mads/rdf/v1#GenreForm,http://www.loc.gov/mads/rdf/v1#Temporal,http://www.loc.gov/mads/rdf/v1#Geographicetc. also the MADSpropertyhttp://www.loc.gov/mads/rdf/v1#isMemberOfMADSScheme. Also some bflc properties:http://id.loc.gov/ontologies/bflc/consolidates,http://id.loc.gov/ontologies/bflc/relatorMatchKey,http://id.loc.gov/ontologies/bflc/applicableInstitutionwhere the first two are also missing in the workshop context (see Some property/type URIs are not compacted #32).Why not use the context at https://github.com/hbz/swib18-workshop/blob/master/data/context.json? (It would be easier for me to work with one context anyway.)
http://I get all fields with a "http" in them. There are those already noted in 1.) and some more that have in common that they have"@language": nullin the context at https://github.com/hbz/lobid-resources/blob/523-addLanguageTagSupport/web/conf/context-loc.jsonldI guess, nearly all errors will be fixed with using the workshop context. The rest will be fixed with #32.
acka47 commentedon Nov 13, 2018
For documentation: Some records are not indexed in ES because they have no valid date as value of
adminMetadata.changeDate.acka47 commentedon Nov 13, 2018
BTW, a lessons learned for Kibana: At "Management" -> "index patterns" -> "loc_works" I had to refresh the field list. Otherwise the fields from the previous index would be shown there and also in the field autosuggests for creating a visualization.
Use workshop's context
Add runner script for loc bibframe
dr0i commentedon Nov 13, 2018
@acka47 I indexed loc-works-bibframe with the workshop's json-context. You have to set the index-alias to use it for kibana. 3k more documents are missing now, you will get tomorrow morning the lobid-admin mail complaining about
[MapperParsingException[object mapping for [language] tried to parse field [null]and some more forsubjectetc. and the knownadminMetadata.changeDate.acka47 commentedon Nov 14, 2018
Taking a look at the new index, some of the property/type URIs are now compacted but some others are still missing:
http://www.loc.gov/mads/rdf/v1#ComplexSubjecthttp://www.loc.gov/mads/rdf/v1#GenreForm-> missing in context as there is alsobf:GenreFormhttp://www.loc.gov/mads/rdf/v1#Temporal-> missing in context as there is alsobf:Temporalhttp://www.loc.gov/mads/rdf/v1#Geographichttp://www.loc.gov/mads/rdf/v1#isMemberOfMADSSchemehttp://id.loc.gov/ontologies/bflc/consolidateshttp://id.loc.gov/ontologies/bflc/relatorMatchKeyhttp://id.loc.gov/ontologies/bflc/applicableInstitutionFor
bflc:consolidatesandbflc:relatorMatchKeysee #32 (comment). I also found the problem withmads:GenreFormandmads:Temporal. What I do not understand is whymads:isMemberOfMADSSchemeis not compacted as it is in the current context, see https://github.com/hbz/swib18-workshop/blob/master/data/context.json#L1114acka47 commentedon Jan 7, 2019
Closing this one as index and Kibana are up. The data may not be perfect but it is totally fine for an experimental setup.