Skip to content

Set up Elasticsearch & openly available kibana with complete Loc-instances-Bibframe dataset #23

Closed

Description

@acka47
No description provided.

Activity

  1. added and removed on Nov 5, 2018
  2. changed the title [-]Set up Elasticsearch & openly available kibana with complete Bibframe dataset[/-] [+]Set up Elasticsearch & openly available kibana with complete Loc-instances-Bibframe dataset[/+] on Nov 5, 2018
  3. dr0i commented on Nov 5, 2018

    @dr0i
    Member

    While the solution provided in a) hbz/lobid-resources#851 is not proper enough but b) #2 not fast enough we go again with a) using java-jsonld, facing some issues which must be resolved, see hbz/lobid-resources#940.

  4. acka47 commented on Nov 9, 2018

    @acka47
    ContributorAuthor

    @dr0i is now able to convert the LOC data with the Java JSON-LD library and the output looks much like what we generate in the workshop. Here is the current status with the same example that we use in the workshop:

    The current differences to the JSON-LD from the workshop (at least those I have found) are:

  5. added a commit that references this issue on Nov 12, 2018
  6. dr0i commented on Nov 12, 2018

    @dr0i
    Member

    Did that. Also indexed a 100k test resources , see curl "http://weywot5.hbz-nrw.de:9200/loc_works/_search?q=*" |json_pp. ETL time was 5 minutes so it takes roughly the same time as the ETL of lobid-resources.

  7. assigned and unassigned on Nov 12, 2018
  8. acka47 commented on Nov 12, 2018

    @acka47
    ContributorAuthor

    Thanks for indexing. Will soon take a look.

    so it takes roughly the same time as the ETL of lobid-resources.

    \o/ This is good to hear.

  9. acka47 commented on Nov 13, 2018

    @acka47
    ContributorAuthor

    Just took a look at the data. Here is what I noticed.

    1. Some things are missing in the context as MADS classes are not compacted: http://www.loc.gov/mads/rdf/v1#ComplexSubject, http://www.loc.gov/mads/rdf/v1#GenreForm, http://www.loc.gov/mads/rdf/v1#Temporal, http://www.loc.gov/mads/rdf/v1#Geographic etc. also the MADSproperty http://www.loc.gov/mads/rdf/v1#isMemberOfMADSScheme. Also some bflc properties: http://id.loc.gov/ontologies/bflc/consolidates, http://id.loc.gov/ontologies/bflc/relatorMatchKey, http://id.loc.gov/ontologies/bflc/applicableInstitution where the first two are also missing in the workshop context (see Some property/type URIs are not compacted #32).
      Why not use the context at https://github.com/hbz/swib18-workshop/blob/master/data/context.json? (It would be easier for me to work with one context anyway.)
    2. Kibana is quite helpful for finding other errors. When I got to "Management" -> "Index Patterns" -> select "loc_works" and then filter by http:// I get all fields with a "http" in them. There are those already noted in 1.) and some more that have in common that they have "@language": null in the context at https://github.com/hbz/lobid-resources/blob/523-addLanguageTagSupport/web/conf/context-loc.jsonld

    I guess, nearly all errors will be fixed with using the workshop context. The rest will be fixed with #32.

  10. acka47 commented on Nov 13, 2018

    @acka47
    ContributorAuthor

    For documentation: Some records are not indexed in ES because they have no valid date as value of adminMetadata.changeDate.

  11. acka47 commented on Nov 13, 2018

    @acka47
    ContributorAuthor

    BTW, a lessons learned for Kibana: At "Management" -> "index patterns" -> "loc_works" I had to refresh the field list. Otherwise the fields from the previous index would be shown there and also in the field autosuggests for creating a visualization.

  12. dr0i commented on Nov 13, 2018

    @dr0i
    Member

    @acka47 I indexed loc-works-bibframe with the workshop's json-context. You have to set the index-alias to use it for kibana. 3k more documents are missing now, you will get tomorrow morning the lobid-admin mail complaining about [MapperParsingException[object mapping for [language] tried to parse field [null] and some more for subject etc. and the known adminMetadata.changeDate.

  13. acka47 commented on Nov 14, 2018

    @acka47
    ContributorAuthor

    Taking a look at the new index, some of the property/type URIs are now compacted but some others are still missing:

    • http://www.loc.gov/mads/rdf/v1#ComplexSubject
    • http://www.loc.gov/mads/rdf/v1#GenreForm -> missing in context as there is also bf:GenreForm
    • http://www.loc.gov/mads/rdf/v1#Temporal -> missing in context as there is also bf:Temporal
    • http://www.loc.gov/mads/rdf/v1#Geographic
    • http://www.loc.gov/mads/rdf/v1#isMemberOfMADSScheme
    • http://id.loc.gov/ontologies/bflc/consolidates
    • http://id.loc.gov/ontologies/bflc/relatorMatchKey
    • http://id.loc.gov/ontologies/bflc/applicableInstitution

    For bflc:consolidates and bflc:relatorMatchKey see #32 (comment). I also found the problem with mads:GenreForm and mads:Temporal. What I do not understand is why mads:isMemberOfMADSScheme is not compacted as it is in the current context, see https://github.com/hbz/swib18-workshop/blob/master/data/context.json#L1114

  14. acka47 commented on Jan 7, 2019

    @acka47
    ContributorAuthor

    Closing this one as index and Kibana are up. The data may not be perfect but it is totally fine for an experimental setup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions