fix(studio-be): fix cache invalidation being discarded by flow service

[laurentlp]

This PR should fix an issue where overriding a bot with the same id (or uploading a bot with an id that already existed) would result in having outdated flow content.

Here is the reason why we should check for matches with and without the prefix /data: botpress/studio/#2

This endpoint (/invalidateFile) is called whenever there is an invalidation in Botpress. Since we remove the data/ part from the key when running using SQLite, the flow-service would discard the cache invalidation resulting in outdated flow data being used.

I also updated the config loader and hint service to make sure that they watch for both keys containing or not the data/ part.

Closes botpress/v12#1581
Closes DEV-2203

Testing

1. To reproduce the issue, create an on_stage_request hook using this code:

  const STAGE_SEPERATOR = '__'
  const VERSION_SEPERATOR = ' :: v'
  const botId = bot.id.split(STAGE_SEPERATOR)[0]
  const STAGE_KEY = `${botId}-PipelineVersion`
  const botLogger = bp.logger.forBot(bot.id)

  const getPromoteVersionStage = () => {
    if (pipeline.length < 2) return 'NEVER'
    let stage2 = pipeline[1]
    return stage2.id
  }

  const getNextVersion = async () => {
    const promoteVersion = getPromoteVersionStage() === bot.pipeline_status.stage_request.id

    let globalKvs = bp.kvs.global()
    let storeExists = await globalKvs.exists(STAGE_KEY)
    let curVersion = storeExists ? await globalKvs.get(STAGE_KEY) : 0

    if (!promoteVersion) return curVersion

    let nextVersion = curVersion + 1

    await globalKvs.set(STAGE_KEY, nextVersion)

    return nextVersion
  }

  const getStageId = () => [botId, bot.pipeline_status.stage_request.id].join(STAGE_SEPERATOR)

  const getStageName = async () => {
    let botName = bot.name.split(VERSION_SEPERATOR)[0]
    let nextVersion = await getNextVersion()

    return `${botName}${VERSION_SEPERATOR}${nextVersion}`
  }

  const handler = async () => {
    try {
      let stageId = getStageId()
      let stageName = await getStageName()

      bot.id = stageId
      bot.name = stageName
    } catch (err) {
      botLogger.error(JSON.stringify(err, null, 2))
    }
  }

  return handler()

2. Make sure you add at least one other stage to your Bot Pipeline.
3. Create an empty bot and add two nodes with content to it.
4. Promote your bot to the next stage.
5. Both bots should work properly.
6. Remove one of the nodes in the original bot.
7. Promote it again
8. This time the promoted bot will still display the deleted node.

You can then repeat these steps to see that the issue is not resolved.

[laurentlp]

@allardy I need more details on why we have to handle this when we call /invalidateFile :

if (process.BPFS_STORAGE === 'disk') {
  key = key.replace('::data/', '::')
}

We might want to move this logic inside a util function so that all services listening to invalidation can select the files they want to handle accordingly

[linear]

DEV-2203 (BUG) Overriding a bot on a stage breaks DialogEngine and fails to update content-elements (botpress/botpress botpress/v12#1581)

Describe the bug
In a workspace with 3 pipeline stages when a bot on some stage is overridden (using on_request_change pipeline hook below):

1. Newly added contents are not visible in the flow editor and contents page of the bot
   
2. Chat stops working
   

May be related to this issue.

To Reproduce
Steps to reproduce the behavior:

1. Set up a pipeline with 3 stages
2. Create an empty bot
   
3. Add some initial content to it
   
4. Promote the bot to the next stage
   
5. Promoted bot should work for now
   
6. Add some new content to the original bot and promote it again
   
7. Open the studio for the promoted bot. You should see newly added content-elements rendered empty and webchat not working
   
   

Expected behavior
The promoted bot should have a working webchat and render newly added content-elements in the flow editor

Environment (please complete the following information):

• OS: macOS
• Browser Chrome
• Browser Version Version 96.0.4664.110 (Official Build) (x86_64)
• Botpress Version v12.26.7
• NodeJS v12.18.1

Additional context
Add any other context about the problem here.

botpress/borpress botpress/v12#1581 by @ daukadolt

[samuelmasse]

LGTM