Merge pull request #1316 from botpress/f_ghost-doc

docs: Versioning documentation

docs/guide/docs/manage/versions.md

@@ -0,0 +1,63 @@
+---
+id: versions
+title: Version control
+---
+
+Once your bot is deployed, the good part is that you (and non-technical team members) **can still make changes to your bots from the Botpress studio**. This is one major advantage of using Botpress. This is made possible by our built-in versioning system.
+
+## Overview
+
+As for now, you probably know that some of your bot's behavior is determined by the content coming from the files (Content, Flows, Actions).
+
+For your convenience Botpress provides the GUI tools to edit these files while in development. We also provide the same tools in production, but there's a caveat. Writing changes to the server's file system is not always possible, they could easily be lost due to the nature of ephemeral server instances. (e.g. when the new version of the bot is deployed, the old server instance may be shut down by the cloud hosting platform).
+
+To address this issue, we give you the Ghost Content feature. In production, your changes are saved to the database which is persisted between deployments. But how do you get these changes back to your bot's codebase? Simple, the botpress cli gives you a special command to pull pending changes on your server for all your bots and server wide files. `./bp pull --url {SERVER_URL} --authToken {YOUR_AUTH_TOKEN} --targetDir {TARGET_DIRECTORY}`
+
+You can also head to the versioning tab of your botpress admin panel at https://your.bp.ai/admin/versioning, the command will be properly formatted for you (including your token) any changes have been made. Just paste it to your shell and the changes will be extracted in the provided target directory. A successful output should look like the following: 
+
+![versioning pull](assets/versioning-pull.png)
+
+
+Notice that without any changes, you will see a **You're all set!** message.
+
+## Workflow
+
+A best practice is to keep the changes of your bots in your prefered Source Control Management tool (e.g Git) and always deploy the master branch in production. Once deployed, you can regularly pull production changes and apply them to your SCM, or revert them at any moment. With this tip you can harness the power of your SCM for branches, merge conflicted files, review changes & create revisions.
+
+Fine, now what if you have a more complex deployment pipeine with a(or multiple) staging environment with pending changes on each enviroment? That's what we'll learn next.
+
+## Pipelines
+
+In this section, we will again use the power of your preferred Source Control Tool (we'll use git in this tutorial) to sync changes between 2 enviroment & promote an enviroment (i.e. promote staging to production).
+
+Given a pipeline with 3 enviroments, **development**, **staging** and **production**. Let's say there are some changes both on production and staging & you want to promote staging to production. What we want to do is the following : 
+
+1- to create merge conflict so we can choose what we want in a merge conflict tool. 
+2- resolve conflicts (i.e merge staging into production)
+3- push the results to master so it can be deployed to your production environment.
+
+That simple. Let's do it.
+
+First, create a branch and sync it with the production enviroment
+
+`git checkout master && git checkout -b prod-sync`
+
+`./bp pull --url {PROD_SERVER_URL} --authToken {YOUR_AUTH_TOKEN} --targetDir {TARGET_DIRECTORY}`
+
+`git commit -am 'sync prod'`
+
+Repeat the process with staging env
+
+`git checkout master && git checkout -b staging-sync`
+
+`./bp pull --url {STAGING_SERVER_URL} --authToken {YOUR_AUTH_TOKEN} --targetDir {TARGET_DIRECTORY}`
+
+`git commit -am 'sync staging'`
+
+Then merge the staging changes into the prod changes
+
+`git checkout prod-sync && git merge staging-sync`
+
+This will create a merge conflict, use your prefered merge tool to review the changes & resolve the conflicts. Once that done, you will be able to publish your branch and create a pull request (if your hosted git allows it) and merge it to master. Once your master branch is up to date, you'll be able to publish the changes in production and we're done.
+
+With these quick tips you can now promote any enviroment changes to any stage in your deployment pipeline.

docs/guide/website/i18n/en.json

@@ -50,6 +50,9 @@
       "manage/sync-changes": {
         "title": "Syncing Changes"
       },
+      "manage/versions": {
+        "title": "Versions"
+      },
       "migrate": {
         "title": "Migrating from 10.X"
       },
@@ -139,6 +142,21 @@
       },
       "version-11.2.0-troubleshoot": {
         "title": "Troubleshoot"
+      },
+      "version-11.3.0-channels": {
+        "title": "Messaging Channels"
+      },
+      "version-11.3.0-code": {
+        "title": "Custom Code"
+      },
+      "version-11.3.0-debug": {
+        "title": "Debugging"
+      },
+      "version-11.3.0-nlu": {
+        "title": "NLU"
+      },
+      "version-11.3.0-overview": {
+        "title": "High-level Overview"
       }
     },
     "links": {
@@ -150,6 +168,7 @@
       "Getting Started": "Getting Started",
       "Build": "Build",
       "Deploy": "Deploy",
+      "Manage": "Manage",
       "Developers": "Developers"
     }
   },

docs/guide/website/sidebars.json

@@ -1,6 +1,11 @@
 {
   "docs": {
-    "Getting Started": ["introduction", "installation", "troubleshoot", "quickstart"],
+    "Getting Started": [
+      "introduction",
+      "installation",
+      "troubleshoot",
+      "quickstart"
+    ],
     "Build": [
       "build/overview",
       "build/debug",
@@ -12,7 +17,17 @@
       "build/channels",
       "build/debugging"
     ],
-    "Deploy": ["deploy/hosting"],
-    "Developers": ["release-notes", "migrate", "create-module", "tutorials"]
+    "Deploy": [
+      "deploy/hosting"
+    ],
+    "Manage": [
+      "manage/versions"
+    ],
+    "Developers": [
+      "release-notes",
+      "migrate",
+      "create-module",
+      "tutorials"
+    ]
   }
-}
+}

package.json

@@ -125,4 +125,4 @@
       "commit-msg": "commitlint -E HUSKY_GIT_PARAMS"
     }
   }
-}
+}

src/bp/core/routers/admin/index.ts

@@ -39,7 +39,7 @@ export class AdminRouter implements CustomRouter {
     this.teamsRouter = new TeamsRouter(logger, this.authService, this.adminService)
     this.usersRouter = new UsersRouter(logger, this.authService, this.adminService)
     this.licenseRouter = new LicenseRouter(logger, this.licenseService)
-    this.versioningRouter = new VersioningRouter(this.adminService, this.ghostService, this.botLoader)
+    this.versioningRouter = new VersioningRouter(this.ghostService, this.botLoader)
 
     this.setupRoutes()
   }

src/bp/core/routers/admin/versioning.ts

@@ -1,19 +1,13 @@
 import { BotLoader } from 'core/bot-loader'
 import { GhostService } from 'core/services'
-import { AdminService } from 'core/services/admin/service'
-import { RequestHandler, Router } from 'express'
+import { Router } from 'express'
 
 import { CustomRouter } from '..'
-import { needPermissions } from '../util'
 
 export class VersioningRouter implements CustomRouter {
   public readonly router: Router
 
-  private _needPermissions: (operation: string, resource: string) => RequestHandler
-
-  constructor(private adminService: AdminService, private ghost: GhostService, private botLoader: BotLoader) {
-    this._needPermissions = needPermissions(this.adminService)
-
+  constructor(private ghost: GhostService, private botLoader: BotLoader) {
     this.router = Router({ mergeParams: true })
     this.setupRoutes()
   }
@@ -23,7 +17,8 @@ export class VersioningRouter implements CustomRouter {
       '/pending',
       // TODO add "need super admin" once superadmin is implemented
       async (req, res) => {
-        res.send(await this.ghost.global().getPending())
+        const botIds = await this.botLoader.getAllBotIds()
+        res.send(await this.ghost.getPending(botIds))
       }
     )
 
@@ -42,13 +37,5 @@ export class VersioningRouter implements CustomRouter {
         res.end(tarball)
       }
     )
-
-    // Revision ID
-    this.router.post('/revert', this._needPermissions('write', 'bot.ghost_content'), async (req, res) => {
-      const revisionId = req.body.revision
-      const filePath = req.body.filePath
-      await this.ghost.forBot(req.params.botId).revertFileRevision(filePath, revisionId)
-      res.sendStatus(200)
-    })
   }
 }

src/bp/core/services/ghost/db-driver.ts

@@ -6,11 +6,11 @@ import { VError } from 'verror'
 import Database from '../../database'
 import { TYPES } from '../../types'
 
-import { GhostFileRevision, StorageDriver } from '.'
+import { FileRevision, StorageDriver } from '.'
 
 @injectable()
 export default class DBStorageDriver implements StorageDriver {
-  constructor(@inject(TYPES.Database) private database: Database) {}
+  constructor(@inject(TYPES.Database) private database: Database) { }
 
   async upsertFile(filePath: string, content: string | Buffer, recordRevision: boolean): Promise<void>
   async upsertFile(filePath: string, content: string | Buffer): Promise<void>
@@ -126,7 +126,7 @@ export default class DBStorageDriver implements StorageDriver {
     }
   }
 
-  async listRevisions(pathPrefix: string): Promise<GhostFileRevision[]> {
+  async listRevisions(pathPrefix: string): Promise<FileRevision[]> {
     try {
       let query = this.database.knex('srv_ghost_index')
 
@@ -138,7 +138,7 @@ export default class DBStorageDriver implements StorageDriver {
       return await query.then(entries =>
         entries.map(
           x =>
-            <GhostFileRevision>{
+            <FileRevision>{
               path: x.file_path,
               revision: x.revision,
               created_on: new Date(x.created_on),

src/bp/core/services/ghost/disk-driver.ts

@@ -4,7 +4,7 @@ import { inject, injectable } from 'inversify'
 import path from 'path'
 import { VError } from 'verror'
 
-import { GhostFileRevision, StorageDriver } from '.'
+import { FileRevision, StorageDriver } from '.'
 
 @injectable()
 export default class DiskStorageDriver implements StorageDriver {
@@ -70,7 +70,7 @@ export default class DiskStorageDriver implements StorageDriver {
     throw new Error('Method not implemented.')
   }
 
-  async listRevisions(pathPrefix: string): Promise<GhostFileRevision[]> {
+  async listRevisions(pathPrefix: string): Promise<FileRevision[]> {
     try {
       const content = await this.readFile(path.join(pathPrefix, 'revisions.json'))
       return JSON.parse(content.toString())

src/bp/core/services/ghost/index.ts

@@ -3,23 +3,24 @@ export interface StorageDriver {
   readFile(filePath: string): Promise<Buffer>
   deleteFile(filePath: string, recordRevision: boolean): Promise<void>
   directoryListing(folder: string, exlude?): Promise<string[]>
-  listRevisions(pathPrefix: string): Promise<GhostFileRevision[]>
+  listRevisions(pathPrefix: string): Promise<FileRevision[]>
   deleteRevision(filePath: string, revision: string): Promise<void>
 }
 
-export type GhostFileRevision = {
+export type FileRevision = {
   path: string
   revision: string
   created_by: string
   created_on: Date
 }
 
-export type GhostPendingRevisions = {
-  [rootFolder: string]: Array<GhostFileRevision>
+export type PendingRevisions = {
+  [rootFolder: string]: Array<FileRevision>
 }
 
-export type GhostPendingRevisionsWithContent = {
-  [rootFolder: string]: Array<GhostFileRevision & { content: Buffer }>
+export interface ServerWidePendingRevisions {
+  global: PendingRevisions
+  bots: PendingRevisions[]
 }
 
 export * from './cache-invalidators'

src/bp/core/services/ghost/service.test.ts

@@ -6,7 +6,7 @@ import 'reflect-metadata'
 import { PersistedConsoleLogger } from '../../logger'
 import { createSpyObject, MockObject } from '../../misc/utils'
 
-import { GhostFileRevision } from '.'
+import { FileRevision } from '.'
 import DBStorageDriver from './db-driver'
 import DiskStorageDriver from './disk-driver'
 import { GhostService } from './service'
@@ -37,8 +37,7 @@ describe('Ghost Service', () => {
     it('DB Driver is never ever called', async () => {
       await ghost.global().deleteFile('', '')
       await ghost.global().directoryListing('', '')
-      await ghost.global().getPending()
-      await ghost.global().getPendingWithContent()
+      await ghost.global().getPendingChanges()
       await ghost.global().isFullySynced()
       await ghost.global().readFileAsBuffer('', '')
       await ghost.global().sync([''])
@@ -102,14 +101,14 @@ describe('Ghost Service', () => {
       dbDriver.listRevisions.mockReturnValue([{ file_path: 'abc', revision: 'rev' }]) // Even if DB driver says there are some revisions
       await ghost.forBot(BOT_ID).upsertFile('test', 'a.json', 'Hello') // And that we modify a file
 
-      const revisions = await ghost.global().getPending()
+      const revisions = await ghost.global().getPendingChanges()
       expect(revisions).toMatchObject({})
     })
   })
 
   describe('Using DB Driver', async () => {
     const buildRev = n =>
-      <GhostFileRevision>{
+      <FileRevision>{
         path: 'file',
         revision: n
       }
@@ -224,7 +223,7 @@ describe('Ghost Service', () => {
     describe('revisions', async () => {
       it('empty when no revisions', async () => {
         dbDriver.listRevisions.mockReturnValue([])
-        const pending = await ghost.global().getPending()
+        const pending = await ghost.global().getPendingChanges()
         expect(Object.keys(pending)).toHaveLength(0)
       })
       it('returns grouped list of revisions when files modified', async () => {
@@ -233,7 +232,7 @@ describe('Ghost Service', () => {
         const r3 = { path: './data/global/b/3.txt', revision: 'r3' }
 
         dbDriver.listRevisions.mockReturnValue([r1, r2, r3])
-        const pending = await ghost.global().getPending()
+        const pending = await ghost.global().getPendingChanges()
 
         expect(Object.keys(pending)).toHaveLength(2)
         expect(pending['a']).toHaveLength(2)
@@ -242,24 +241,6 @@ describe('Ghost Service', () => {
         expect(pending['a'][0].path).toContain('1.txt')
         expect(pending['a'][0].revision).toContain('r1')
       })
-
-      it('revision with content works', async () => {
-        const r1 = { path: './data/global/a/1.txt', revision: 'r1' }
-        const r2 = { path: './data/global/a/2.txt', revision: 'r2' }
-        const r3 = { path: './data/global/b/3.txt', revision: 'r3' }
-
-        dbDriver.readFile.mockReturnValue(Buffer.from('content'))
-        dbDriver.listRevisions.mockReturnValue([r1, r2, r3])
-
-        const pending = await ghost.global().getPendingWithContent()
-
-        expect(Object.keys(pending)).toHaveLength(2)
-        expect(pending['a']).toHaveLength(2)
-        expect(pending['b']).toHaveLength(1)
-
-        expect(pending['a'][0].content).toBeDefined()
-        expect(pending['a'][0].content.toString()).toBe('content')
-      })
     })
   })
 })