New catalog-solr instance is using wrong username/password

[FuhuXia]

New Solr instances created by scaling up catalog-solr are using the default solr username password. As a result, catalog-web gives solr error when it is connected to the new solr instances. Scale back down to the previous level to avoid the solr error. This issue means our solr followers are stuck with the current scale as the maximum instances we can have.

How to reproduce

1. increase solrFollowerCount in the catalog app solr/service-config.json
2. run cf update-service catalog-solr -c solr/service-config.json --wait
3. check catalog.

Expected behavior

No solr error

Actual behavior

Solr error when the new instance is used.

Sketch

[Notes or a checklist reflecting our understanding of the selected approach]

[nickumia]

@FuhuXia Is this for an old solr instance? It sounds like it's running any old version of the admin startup script. I would check the version of the broker used to create/update the solr instance. I would also check the the script on the solr-admin-init container to see how many solr instances are there... The best answer is probably that the solr-admin-init container needs to be restarted (if that was forgotten) or if that manual step is not good enough, the brokerpak could be updated.

The password doesn't get set if the solr-admin-init container doesn't update the security.json. If you need more to spur the memory (or if I'm completely wrong), let me know and we can talk more.

[FuhuXia]

Gave it a shot today, increased replica count before and after a solr-admin-init task restart. Did not resolved the issue. Will look into the logs and solr-admin-init health status.

[nickumia]

When you check the logs (I would try to see if you can see the plan that the brokerpak ran and make sure that the solr-admin-init task definition was actually updated..

• This is the list of solr instances that it uses to update the password: https://github.com/GSA-TTS/datagov-brokerpak-solr/blob/main/terraform/ecs/provision/admin.tf#L10
• This is where it actually updates the password: https://github.com/GSA-TTS/datagov-brokerpak-solr/blob/main/terraform/ecs/provision/admin.tf#L67-L70

I'm now remembering that I may have manually edited the task definition in the AWS Console (if we never fixed the bug in the brokerpak that causes this). If you edit the task definition (with the correct number of solr instances) and then make sure to start a new container based on the new task definition, that will definitely be the "workaround". I assume that this isn't a common thing that we were going to support (changing the number of followers, but the long-term fix would be making sure the admin-task-definition gets tainted and recreated when the follower_count changes).

[FuhuXia]

Here is what we know so far:

1. Restarting solr-admin-init then scaling up does not help
2. Scaling up on a new solr cluster has the same issue
3. Starting a new solr cluster with desired follower count works. This is the workaround so far.

The # 3 workaround requires rebuilding solr index, which is time-consuming. Will look at @nickumia's idea on editing task definition for a better workaround.

[FuhuXia]

The issue arises because the solr-admin-init task definition, which is established during service creation, has a fixed number of followers to which it assigns credentials. This number remains static, regardless of whether the followers are scaled up or down. Consequently, any followers added beyond the initial number will not receive credentials from the solr-admin-init. A more effective workaround would be to modify the solr-admin-init task definition script to iterate over a larger number of followers, for example, 10. This way, any scaling up or down of followers within this limit will ensure proper credential assignment.