feat: Index key generation related fixes

[chandrasekharan-zipstack]

What

• Added migration to delete incorrect index keys
• Updated index key generation in prompt service
• MINOR: Corrected the /document-index request to include a trailing slash and avoid an unnecessary redirect
• Version and lockfile updates for tools and other services to use SDK 0.24.0

Why

• More context in SDK PR

Can this PR break any existing features. If yes please list of possible items. If no please explain why. (PS: Admins do not merge the PR without this section filled)

• Yes, its related to an SDK PR. Since there's an update in the index key generated - all old index keys would be invalid. In order to perform a clean up - a migration has been added

Database Migrations

• Simple migration - clearing all old index key records only if they matched the regex from the old logic. This would ensure no new records get deleted unnecessarily if migrations are re-run

Env Config

Relevant Docs

Related Issues or PRs

• feat: Index key generation fix unstract-sdk#42

Dependencies Versions

Notes on Testing

• Explicitly checked DB before and after migrations
• Also tried to update the adapters after it was indexed once to ensure it was indexed again

Screenshots

Checklist

I have read and understood the Contribution Guidelines.

[hari-kuriakose]

@chandrasekharan-zipstack Data migration looks ok.

However let's test the migration once before we merge this in.

[chandrasekharan-zipstack]

@hari-kuriakose regarding this - in the current system as well we don't remove the stale entries. I thought we could handle all of it together with a separate task to delete these vector DB entries when indexes are invalidated.
But now that you mention it we might end up with stray records in the vector DB for these keys that will be deleted - we might have to loop over all the vector DB instances which was used to index and then invoke some delete method to do this. In the interest of time, we decided to ignore these entries alone
CC: @nehabagdia @tahierhussain

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud