Index Alma data triggered by webhook: daily updates and basedump

[dr0i]

Complements #1112 .
Fundamental to solve #1162.

@blackwinter has implemented a webhook which would call a (to be implemented) webhook listener to inform it about when the updates/basedump files are ready.

• implement a webhook listener
• index the data when webhook listener gets triggered
• put webhook listener into monit to ensure its living (as it is part of lobid's web app it is already monitored)
• get SSL certificate
• restrict to SSL
• configure webhook listener in the webhook

Deletions are atm not important but would complement this issue.

[blackwinter]

There are two possible webhooks (per type of run):

1. baseline: Gets triggered when a baseline run finishes.
2. update: Gets triggered when an update run finishes.

Both URLs may contain the following optional placeholders:

• %{job}: The job to trigger (in our case: Jenkins job name); configurable per type of run (see above).
• %{token}: A secret to protect the listener from unauthorized access (in our case: Jenkins Authentication Token); configurable per type of run (see above).
• %{cause}: Some metadata pertaining to the run triggering the webhook; dynamic per run.

For Jenkins, our URL would currently look like:

http://JENKINS_HOST/job/%{job}/build?token=%{token}&cause=%{cause}

[dr0i]

Webhook listener deployed, called like:
http://alma.lobid.org/resources/webhook/update-alma?token=$secret and
http://alma.lobid.org/resources/webhook/basedump-alma?token=$secret.
Waiting for @blackwinter to configure the listener in their Webhook.

[dr0i]

Uh, wait: we should stick to SSL because of the secret token. Will assign @blackwinter again if the https URIs are properly configured.

[blackwinter]

The webhook endpoint should only enqueue an indexing job, not run it immediately (at least not synchronously). We're not going to wait for the indexing process to finish. If you can't/won't adjust your webhook listener, we'd need to change the trigger to avoid waiting for a response.

[dr0i]

Ah, yes I see, I did that wrong. Will fix this so that the endpoint is going to immediately ack when triggered.

[dr0i]

Ups, hadn't deployed. Shouldn't do something quickly before lunch .... Please try again.

[dr0i]

(the multiple started processes seem to be in connection with update.xml.bgzf beeing only 122 bytes small, resulting in just a little "locked" status. If you call the listener fast enough, it should work).

[dr0i]

Hm, something is strange and I cannot see the logs ETL program. Have to check.

[blackwinter]

Let me know if/when I shall give it another try. It's configured now and will be included when publishing starts again next week.

[dr0i]

Got logging to work (had to copy the log4j.xml to web/conf). Give it another try.

[blackwinter]

LGTM.

[dr0i]

Thx for reviewing and configuring! Closing.

[dr0i]

Uh. There was a bug in the creation of the index name, resulting in an overwriting of the existing production index. This happened 2 hours ago with the Webhook listener triggered from the BGFZ creation after NZ publishing.
Switched back to an index pre #1214 (Thursday last week) , fixed the bug in 5caa61d and restarted deploying new index.

[dr0i]

ETLed the basedump of today and made it productive. Closing.