Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not block startup if typesense is down #1525

Merged
merged 1 commit into from Aug 30, 2021
Merged

Do not block startup if typesense is down #1525

merged 1 commit into from Aug 30, 2021

Conversation

amCap1712
Copy link
Member

Typesense was acting up today and needed to be shutdown. This brought down LB along with it. In case we need to shutdown typesense again, disabling startup improvements for it. The SERVICEDOESNOTEXIST sentinel is grepped by the startup script to detect issues and prevent container from starting. Setting it to any other value so that container starts even in absence of typesense.

@amCap1712
Copy link
Member Author

No need to merge this, I have pushed an image based on this branch metabrainz/listenbrainz:no-typesense in case we need to deploy this.

@amCap1712
Copy link
Member Author

Typesense is fixed now, closing.

@amCap1712 amCap1712 closed this Jun 25, 2021
@amCap1712 amCap1712 reopened this Jul 20, 2021
@amCap1712 amCap1712 closed this Jul 22, 2021
@amCap1712 amCap1712 reopened this Aug 10, 2021
@amCap1712
Copy link
Member Author

@alastair investigated using docker health checks and stuff to fix this properly but he found that the consul stuff may change soon so its probably not worth doing just presently. We can relook using health checks in future again and fixing this properly at that time.

Copy link
Collaborator

@alastair alastair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine. I see that the only thing that uses this value is labs api and the mapping, so it doesn't matter what the value is within the LB container.

As a future work, I wonder if something like this should be moved out of the main config file. How would we do this? Flask can load multiple config files - we could say that labs-api loads the main config and then an additional one that contains labs-specific config items? Not sure if this is a good idea, or if it will cause confusions.

@amCap1712
Copy link
Member Author

That sounds like a neat idea and we already do it for cron. But yeah, it may also cause confusion.

@amCap1712 amCap1712 merged commit d8a465e into master Aug 30, 2021
@amCap1712 amCap1712 deleted the typesense branch August 30, 2021 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants