[api] Handle database data changes with a gem.

Use the data-migrate gem to handle changes to data with their own set of
migrations just like we handle changes to the database structure.

This gives us a set structure to update existing data in the database
and gives us a single command to run (rake db:migrate:with_data) in
deployments.

.gitignore

@@ -38,7 +38,6 @@
 /src/api/mkmf.log
 /src/api/tmp
 /src/api/spec/examples.txt
-/src/api/db/data/
 /src/backend/blib
 /src/backend/BSConfig.pm
 /src/backend/BSSolv.bs

ReleaseNotes-2.9

@@ -23,11 +23,6 @@ OBS Appliance users who have setup their LVM can just replace
 their appliance image without data loss. The migration will
 happen automatically.
 
-Upgrading from previous versions:
-=================================
-
- * Please run this rake task if you notifications table has any rows in it:
- rake db:convert_notifications_serialization
 
 Features
 ========

dist/README.UPDATERS

@@ -20,7 +20,7 @@ Note: Update from OBS 2.5 should also work, but is untested.
   4) Migrate database
 
         cd /srv/www/obs/api/
-        RAILS_ENV="production" rails.ruby2.4 db:migrate
+        RAILS_ENV="production" rails.ruby2.4 db:migrate:with_data
 
   5) Make sure that log and tmp are owned by wwwrun
 
@@ -147,7 +147,7 @@ For Updaters to OBS 2.6 from OBS 2.5
 Appliance users can just do a package update and reboot or use the
 new appliance image to update their configurations.
 
-In case the installation is not done via packages all 
+In case the installation is not done via packages all
 rubygems according to Gemfile need to get installed.
 
  1) Database migration
@@ -173,7 +173,7 @@ For Updaters to OBS 2.5 from OBS 2.4
 Appliance users can just do a package update and reboot or use the
 new appliance image to update their configurations.
 
-In case the installation is not done via packages all 
+In case the installation is not done via packages all
 rubygems according to Gemfile need to get installed.
 
 Biggest change setup wise is the unification of API and WEBUI instance.
@@ -195,7 +195,7 @@ use port 443 now.
  2) Restart memcached after database migration to wipe possible broken
     cache data.
 
- 3) configuration settings moved from various palces to unique /configuration 
+ 3) configuration settings moved from various palces to unique /configuration
     api route as much as possible to allow a uniq configuration through all
     OBS frontend and backend parts. The following settings have been migrated:
 
@@ -229,7 +229,7 @@ use port 443 now.
  4) api interface configuration must be configured in /srv/www/obs/api/config/options.yml
     now. Ensure that at least the following parameters are configured there:
 
-      frontend_host: 
+      frontend_host:
       frontend_port:
       frontend_protocol:
 
@@ -249,15 +249,15 @@ Other installations need merge manually configurations
 as described below (This is required due to the Rails 2.x
 to Rails 3.x switch).
 
-In case the installation is not done via packages all 
+In case the installation is not done via packages all
 rubygems according to Gemfile need to get installed.
 
 Please check the following files:
 
  1) database driver needs to be changed to "mysql2" in /srv/www/obs/api/config/database.yml
     This is usally done automatically by the package.
 
- 2) all config options with capital letters have been moved from 
+ 2) all config options with capital letters have been moved from
       /srv/www/obs/.../config/environments/production.rb
     to small letter variables in
       /srv/www/obs/.../config/options.yml
@@ -296,7 +296,7 @@ For Updaters to OBS 2.3 from OBS 2.1
 
    The default httpd is apache2 since OBS 2.3. lighttpd should still work, but
    we recommend to switch to apache to get a maintained base and load
-   optimizations. 
+   optimizations.
 
    Please read the section 3 from README.SETUP file to learn how to configure
    apache.
@@ -339,10 +339,10 @@ For Updaters to OBS 2.1 from OBS 2.0
       mysql> quit
 
    * Configure your MySQL user and password in the "production:" section
-     of the webui config: 
+     of the webui config:
 
         /srv/www/obs/webui/config/database.yml
-    
+
      A template for this file can be found in same directory as "database.yml.example".
 
    * populate the database
@@ -359,7 +359,7 @@ NOTE: Do not update from the special MeeGo 1.8 release yet. You will miss
       features and run into database migration errors.
       You need to wait for OBS 2.1.
 
-After running the package update to obs-* 2.0 packages you need to do the 
+After running the package update to obs-* 2.0 packages you need to do the
 following steps manually:
 
 1) Database migration
@@ -375,7 +375,7 @@ following steps manually:
 2) Default distribution configuration
    The default targets are now defined in one place on the api.
    Create the default config via
- 
+
     # cd /srv/www/obs/api/files/
     # cp distributions.xml.template distributions.xml
 
@@ -412,13 +412,13 @@ For Updaters from OBS 1.6
 
 3) Enable the new services
 
-  In case you use the package signing, you need to run the new bs_signer 
+  In case you use the package signing, you need to run the new bs_signer
   process. This has been split out of bs_repserver for better scalability.
 
     # insserv obssigner
     # rcobs_signer start
 
-  You may want to run also the new bs_warden daemon, this monitors the 
+  You may want to run also the new bs_warden daemon, this monitors the
   build hosts and restarts jobs, in case a build host is not responding
   correctly anymore:
 
@@ -450,7 +450,7 @@ For Updaters from OBS 0.9.x releases
 1) Define download URL in rails config.
 
 The new introduced "DOWNLOAD_URL" need to be defined in your settings
-below /srv/www/obs/webui/config/environments/ directory. 
+below /srv/www/obs/webui/config/environments/ directory.
 If you do not have a download server, simply define
 
 DOWNLOAD_URL = nil
@@ -459,7 +459,7 @@ in your configuration.
 
 2) Recommended rails migrate update
 
-A rails migrate is recommended, but not neccessary. It would reimport 
+A rails migrate is recommended, but not neccessary. It would reimport
 buginfo flags from the backend to fix broken usage in api before.
 
 When the obs-api package has been updated, it might be required to

src/api/Gemfile

@@ -68,6 +68,8 @@ gem 'peek-mysql2'
 gem "gssapi", require: false
 # for sending events to rabbitmq
 gem 'bunny'
+# for making changes to existing data
+gem 'data_migrate'
 
 group :development, :production do
   # to have the delayed job daemon

src/api/Gemfile.lock

@@ -106,6 +106,8 @@ GEM
     cssmin (1.0.3)
     daemons (1.2.4)
     dalli (2.7.6)
+    data_migrate (3.2.1)
+      rails (>= 4.0)
     database_cleaner (1.6.1)
     delayed_job (4.1.3)
       activesupport (>= 3.0, < 5.2)
@@ -430,6 +432,7 @@ DEPENDENCIES
   cssmin (>= 1.0.2)
   daemons
   dalli
+  data_migrate
   database_cleaner (>= 1.0.1)
   delayed_job_active_record (>= 4.0.0)
   escape_utils

src/api/db/data/20170831143534_convert_notifications_event_payload_to_json.rb

@@ -0,0 +1,55 @@
+class ConvertNotificationsEventPayloadToJson < ActiveRecord::Migration[5.1]
+  def self.up
+    Notification20170831143534.transaction do
+      Notification20170831143534.all.find_each do |notification|
+        json = yaml_to_json(notification.event_payload)
+        notification.update_attributes!(event_payload: json)
+      end
+    end
+  end
+
+  def self.down
+    raise ActiveRecord::IrreversibleMigration
+  end
+end
+
+def yaml_to_json(yaml)
+  YAML.safe_load(yaml)
+      .traverse do |value|
+        if value.is_a? String
+          value.force_encoding('UTF-8')
+        else
+          value
+        end
+      end
+      .to_json
+end
+
+# Notification model only for migration in order to avoid errors coming from the serialization in the actual Notification model
+class Notification20170831143534 < ::ApplicationRecord
+  self.table_name = 'notifications'
+  self.inheritance_column = :_type_disabled
+end
+
+# Hash extension is used to run force_encoding against each string value in the hash
+# in data migration to convert yaml to json serialisation for event payloads
+class Hash
+  def traverse(&block)
+    traverse_value(self, &block)
+  end
+
+  private
+
+  def traverse_value(value, &block)
+    if value.is_a? Hash
+      value.each { |key, sub_value| value[key] = traverse_value(sub_value, &block) }
+
+    elsif value.is_a? Array
+      value.map { |element| traverse_value(element, &block) }
+
+    else
+      yield(value)
+
+    end
+  end
+end

src/api/db/structure.sql

@@ -374,6 +374,11 @@ CREATE TABLE `configurations` (
   PRIMARY KEY (`id`)
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
 
+CREATE TABLE `data_migrations` (
+  `version` varchar(255) NOT NULL,
+  UNIQUE KEY `unique_data_migrations` (`version`)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+
 CREATE TABLE `delayed_jobs` (
   `id` int(11) NOT NULL AUTO_INCREMENT,
   `priority` int(11) DEFAULT '0',

src/api/lib/tasks/databases.rake

@@ -1,5 +1,3 @@
-require_relative '../../app/models/application_record'
-
 module Rake
   module TaskManager
     def redefine_task(task_class, *args, &block)
@@ -123,54 +121,11 @@ namespace :db do
     Rake::Task["db:seed"].invoke
   end
 
-  desc "Convert existing notifications to use JSON serialization for the event_payload column"
-  task convert_notifications_serialization: :environment do
-    NotificationForRakeTask.transaction do
-      NotificationForRakeTask.all.find_each do |notification|
-        json = yaml_to_json(notification.event_payload)
-        notification.update_attributes!(event_payload: json)
-      end
-    end
-  end
-end
-
-def yaml_to_json(yaml)
-  YAML.safe_load(yaml)
-      .traverse do |value|
-        if value.is_a? String
-          value.force_encoding('UTF-8')
-        else
-          value
-        end
-      end
-      .to_json
-end
-
-# Notification model only for migration in order to avoid errors coming from the serialization in the actual Notification model
-class NotificationForRakeTask < ::ApplicationRecord
-  self.table_name = 'notifications'
-  self.inheritance_column = :_type_disabled
-end
-
-# Hash extension is used to run force_encoding against each string value in the hash
-# in rake tasks to convert yaml to json serialisation for event payloads
-class Hash
-  def traverse(&block)
-    traverse_value(self, &block)
-  end
-
-  private
-
-  def traverse_value(value, &block)
-    if value.is_a? Hash
-      value.each { |key, sub_value| value[key] = traverse_value(sub_value, &block) }
-
-    elsif value.is_a? Array
-      value.map { |element| traverse_value(element, &block) }
-
-    else
-      yield(value)
-
-    end
+  desc 'Migrate the database (options: VERSION=x, VERBOSE=false, SCOPE=blog)'
+  task migrate: :environment do
+    puts ''
+    puts 'warning: db:migrate only migrates your database structure, not the data contained in it.'
+    puts 'warning for migrating your data run data:migrate'
+    puts ''
   end
 end

src/api/spec/lib/tasks/databases_spec.rb → src/api/spec/db/data/convert_notifications_event_payload_to_json_spec.rb

@@ -1,17 +1,17 @@
 require 'rails_helper'
+require Rails.root.join('db/data/20170831143534_convert_notifications_event_payload_to_json.rb')
 
-RSpec.describe 'rake db' do
-  describe '#convert_notifications_serialization' do
+RSpec.describe ConvertNotificationsEventPayloadToJson do
+  describe '.up' do
     let!(:yaml) { "---\nhello: world\nhow:\n- are\n- you\n- today?\nim: fine thanks\n" }
     let!(:notification) { create(:notification, type: 'Notification::RssFeedItem') }
-    let!(:task) { Rake::Task['db:convert_notifications_serialization'] }
 
     before do
       sql = "UPDATE `notifications` SET `event_payload` = '#{yaml}' WHERE id = #{notification.id}"
       ActiveRecord::Base.connection.execute(sql)
     end
 
-    subject! { task.execute }
+    subject! { ConvertNotificationsEventPayloadToJson.up }
 
     it 'converts the notifications event_payload from yaml to json' do
       json_hash = { "hello" => "world", "how" => ["are", "you", "today?"], "im" => "fine thanks"}

src/api/spec/spec_helper.rb

@@ -80,10 +80,6 @@
       config.formatter = 'NyanUnicornFormatter'
     end
   end
-
-  config.before(:suite) do
-    Rails.application.load_tasks
-  end
 end
 
 # We never want the backend to autostart itself...