Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Solr database delete/load for name-lookup #609

Open
Tracked by #49
gaurav opened this issue Mar 17, 2023 · 2 comments
Open
Tracked by #49

Improve Solr database delete/load for name-lookup #609

gaurav opened this issue Mar 17, 2023 · 2 comments
Assignees

Comments

@gaurav
Copy link
Collaborator

gaurav commented Mar 17, 2023

The current sequence order is:

  1. Init container: We wipe the Solr data directory (which is also where we download the backup)
  2. Init container: We upload the Solr backup into the Solr data directory.
  3. Init container: We start Solr.
  4. Restore-job pod: Command Solr to load the Solr backup into its data directory.

However, this can cause a problem if the Kubernetes instance is restarted -- the init container will wipe the Solr data directory, but the restore-job pod is not triggered.

A better system might be:

  1. Move the deletion/clear operation to the Restore-job pod.
  2. Modify the download code so that the file isn't redownloaded if it hasn't been modified.
@gaurav
Copy link
Collaborator Author

gaurav commented Jun 16, 2023

@YaphetKG has added a way to do this: by running helm upgrade name-lookup ... --set dataUrl='' once name-lookup has started. This ensures that the Solr database is not re-downloaded if the node is restarted. Now the next question is how we can ensure that ITRB does this -- can we integrate this into https://github.com/helxplatform/translator-devops/blob/develop/jenkins/name-lookup/Jenkinsfile somehow? Or maybe just let them to do this when upgrading a node, and see if they have ideas about automating this?

@gaurav gaurav self-assigned this Jun 16, 2023
@gaurav
Copy link
Collaborator Author

gaurav commented Jun 16, 2023

Here's what the configuration looks like in plater, but it's unclear how exactly this parameter is used:

choice(name: 'LOAD_DATA', choices: 'no\nyes', description: 'Load data or not, default no')

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant