Configure scheduled backup

[bobstarbird]

To configure a scheduled backup:

Edit *.properties:

backup-directory=\MarkLogic\Backups\
journal-archive-path=E:\MarkLogic\Journal-Archive\

Edit: app_specific.rb

  def delete_scheduled_backups()
    r = execute_query %Q{
      xquery version "1.0-ml";
      import module namespace admin = "http://marklogic.com/xdmp/admin" 
          at "/MarkLogic/admin.xqy";

      let $config := admin:get-configuration()
      let $database-id :=  xdmp:database("#{@properties['ml.content-db']}")
      let $backups := admin:database-get-backups($config, $database-id)
      let $del-back := admin:database-delete-backup($config, $database-id, $backups)

      return  
        admin:save-configuration($del-back)

      (: Deletes backups. :)
    },
    { :db_name => @properties["ml.content-db"] }
  end

  def create_scheduled_backups()
    r = execute_query %Q{
      xquery version "1.0-ml";
      import module namespace admin = "http://marklogic.com/xdmp/admin" 
          at "/MarkLogic/admin.xqy";
      declare namespace db = "http://marklogic.com/xdmp/database";

      let $config := admin:get-configuration()
      let $database-id :=  xdmp:database("#{@properties['ml.content-db']}")
      let $backups := #{get_config}/db:databases/db:database[db:database-name="#{@properties['ml.content-db']}"]/db:database-backups/db:database-backup
      (:
      let $backups := admin:database-daily-backup(
        "/backup-dir",
        1,
        xs:time("02:00:00"),
        7,
        true(),
        true(),
        true(),
        false())
      :)
      let $create-back := admin:database-add-backup($config, $database-id, $backups)

      return  
        admin:save-configuration($create-back)

      (: Creates backups. :)
    },
    { :db_name => @properties["ml.content-db"] }
  end

Also edit ml-config.xml:

  <database-backups>
    <database-backup>
      <backup-directory>@ml.backup-directory</backup-directory>
      <backup-type>daily</backup-type>
      <backup-period>1</backup-period>
      <backup-days/>
      <backup-start-time>02:00:00-04:00</backup-start-time>
      <max-backups>7</max-backups>
      <backup-security-database>true</backup-security-database>
      <backup-schemas-database>true</backup-schemas-database>
      <backup-triggers-database>true</backup-triggers-database>
      <include-replicas>true</include-replicas>
      <journal-archiving>true</journal-archiving>
      <journal-archive-path>@ml.journal-archive-path</journal-archive-path>
      <journal-archive-lag-limit>900</journal-archive-lag-limit>
    </database-backup>
  </database-backups>

And finally if necessary edit app_specific.rb to create directories

require 'fileutils'

  def make_dirs()
    # FileUtils.mkdir_p("#{@properties['ml.forest-data-dir']}")
    FileUtils.mkdir_p("#{@properties['ml.backup-directory']}")
    FileUtils.mkdir_p("#{@properties['ml.journal-archive-path']}")
  end

[RobertSzkutak]

Got a request from a customer who is currently using this to merge it into our DEV branch sooner than later.

[grtjn]

There are a few things that need to be improved:

• Keep in mind that the backup dirs are on the server, so you cannot use local Ruby code to create them. Maybe it is possible using xdmp:filesystem-directory-create on the server, but likely it needs to be run on all hosts, so not just one execute_query. (Does rest api make this easier?)
• There are db-level backups, but also forest-level backups. Should support both.
• We have bad experience feeding literal pieces of ml-config to admin functions. We should use admin functions to create the actual schedule configs, even though it looks overkill.

Question: do we want to include deploying backups in bootstrap, or make that a separate command. I'm thinking if we make a separate deploy backups, you can do a separate clean backups too. Maybe even do something like ./ml env enable/disable backups. It would give easier control over when to deploy/activate them, e.g. on qa for brief testing (and disabling), and on prod, but not on local dev machines..

[grtjn]

As a consequence, this will be a slightly bigger effort. Keeping this for next milestone..

[ezbc]

@grtjn @RobertSzkutak I am working on supporting parts of this enhancement for a project. Would either of you be interested in discussing how you would like to see this implemented? What do you see as the minimum requirements for supporting a scheduled backup feature in roxy?

My first pass at implementing this is to add a step in "setup:configure-databases" in the setup.xqy which adds scheduled backup configs to the admin configuration. This function would handle the logic for which cadence of backup to add, e.g., daily, monthly, and whether to back up a forest or database.

Depending on what you require, we could implement the features @grtjn mentioned on May 31 in phases. For the first phase we could assume the directories exist on the server, and only support deploying a scheduled backup, without enable or disable functionality.

[bobstarbird]

One not so obvious aspect of backups is the unique directory requirement across databases and retention rules requiring deep directory structures. Maybe even a convention like (/backup/{cadence}_{type}/{dbname} ex: /backups/daily_full/Security, /backups/Weekly_full/Security. I would be good to include separate backups for Security, Schema Triggers because that is our recommendation for production. After the first pass it would be good to have the delete/recreate ability rather than just add.

…

On Fri, Sep 29, 2017 at 10:53 AM, Elijah Bernstein-Cooper < ***@***.***> wrote: @grtjn <https://github.com/grtjn> @RobertSzkutak <https://github.com/robertszkutak> I am working on supporting parts of this enhancement for a project. Would either of you be interested in discussing how you would like to see this implemented? What do you see as the minimum requirements for supporting a scheduled backup feature in roxy? My first pass at implementing this is to add a step in "setup:configure-databases" in the setup.xqy which adds scheduled backup configs to the admin configuration. This function would handle the logic for which cadence of backup to add, e.g., daily, monthly, and whether to back up a forest or database. Depending on what you require, we could implement the features @grtjn <https://github.com/grtjn> mentioned on May 31 in phases. For the first phase we could assume the directories exist on the server, and only support deploying a scheduled backup, without enable or disable functionality. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#196 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABxvG5iCQ1XMRPnVSnE0lzx-aNYiDMuHks5snQRdgaJpZM4BwtUC> .

-- Robert Starbird Software Professional Services Cell: (781) 548-9427 robert.starbird@gmail.com http://www.linkedin.com/in/robertstarbird