The PostgreSQL is dead, long live the SQLite !

[kinoppyd]

What I did in this Pull Request, and how to move the development database to SQLite

Back up the Postgres database with pg_dump

pg_dump postgres://postgres:postgres@127.0.0.1:25432/mie_development --inserts --data-only -T schema_migrations -T ar_internal_metadata -T '*solid_queue*' > tmp/database_backup.sql

• Set the --insert option to dump as SQL INSERT statements instead of using the COPY command.
• Set the --data-only option to exclude table creation queries from the dump file.
• Set the -T schema_migrations and -T ar_internal_metadata options to exclude the ActiveRecord metadata records. They already exists in the SQLite database file when migrating.
• (Optional. if you set this option, you must follow the section on isolating solid_queue database) Set the -T '*solid_queue*' option to exclude the SolidQueue data.
  • Additionally, you must run an additional command to dump the SolidQueue data into another dump file.
  • pg_dump postgres://postgres:postgres@127.0.0.1:25432/mie_development --inserts --data-only -t '*solid_queue*' > tmp/database_solid_queue_backup.sql

Modify the dump file so that it can be restored in SQLite.

• Remove SET statements.
• Remove PostgreSQL table schema.
• Remove pgcatalog.setval statements.

sed -i '' -e 's/public\.//' tmp/database_backup.sql
sed -i '' '/pg_catalog.setval/d' tmp/database_backup.sql

# if you dump SolidQueue data into another dump file.
sed -i '' -e 's/public\.//' tmp/database_solid_queue_backup.sql
sed -i '' '/pg_catalog.setval/d' tmp/database_solid_queue_backup.sql

Remove the pg gem from you Gemfile, and add the sqlite3 gem.

vim Gemfile # Remove the `pg` gem manually in an editor.
bundle add sqlite3

diff --git a/Gemfile b/Gemfile
index a882103..95b7d12 100644
--- a/Gemfile
+++ b/Gemfile
@@ -9,8 +9,8 @@ gem 'rails', '~> 8.0'
 
 gem 'propshaft'
 
-# Use postgresql as the database for Active Record
-gem 'pg', '~> 1.5'
+gem "sqlite3", "~> 2.5"
+
 # Use Puma as the app server
 gem 'puma', '~> 5.0'
 # Use Active Model has_secure_password
diff --git a/Gemfile.lock b/Gemfile.lock
index 9e34e13..1920833 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -218,7 +218,6 @@ GEM
     parser (3.3.6.0)
       ast (~> 2.4.1)
       racc
-    pg (1.5.9)
     prettier_print (1.2.1)
     propshaft (1.1.0)
       actionpack (>= 7.0.0)
@@ -328,6 +327,11 @@ GEM
       fugit (~> 1.11.0)
       railties (>= 7.1)
       thor (~> 1.3.1)
+    sqlite3 (2.5.0)
+      mini_portile2 (~> 2.8.0)
+    sqlite3 (2.5.0-arm64-darwin)
+    sqlite3 (2.5.0-x86_64-darwin)
+    sqlite3 (2.5.0-x86_64-linux-gnu)
     stimulus-rails (1.3.4)
       railties (>= 6.0.0)
     stringio (3.1.2)
@@ -383,7 +387,6 @@ DEPENDENCIES
   mission_control-jobs (~> 1.0)
   omniauth-github (~> 2.0)
   omniauth-rails_csrf_protection (~> 1.0)
-  pg (~> 1.5)
   propshaft
   puma (~> 5.0)
   rack-contrib (~> 2.5.0)
@@ -395,6 +398,7 @@ DEPENDENCIES
   selenium-webdriver
   solid_cache (~> 1.0)
   solid_queue (= 1.1.3)
+  sqlite3 (~> 2.5)
   stimulus-rails (~> 1.3)
   tailwindcss-rails (~> 2.0)
   turbo-rails (~> 2.0)

Modify config/database.yml to use SQLite

diff --git a/config/database.yml b/config/database.yml
index 718be44..bb7d0ea 100644
--- a/config/database.yml
+++ b/config/database.yml
@@ -15,22 +15,19 @@
 # gem 'pg'
 #
 default: &default
-  adapter: postgresql
-  encoding: unicode
+  adapter: sqlite3
   # For details on connection pooling, see Rails configuration guide
   # https://guides.rubyonrails.org/configuring.html#database-pooling
   pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
+  timeout: 5000
 
 development:
   primary: &primary_development
     <<: *default
-    database: mie_development
-    host: "127.0.0.1"
-    user: postgres
-    port: 25432
+    database: storage/development.sqlite3
   cache:
     <<: *primary_development
-    database: mie_development_cache
+    database: storage/development_cache.sqlite3
     migrations_paths: db/cache_migrate
 
   # The specified database role being used to connect to postgres.
@@ -65,10 +62,7 @@ development:
 # Do not set this db to the same as development or production.
 test:
   <<: *default
-  database: mie_test
-  host: "127.0.0.1"
-  user: postgres
-  port: 25432
+  database: storage/test.sqlite3
 
 # As with config/credentials.yml, you never want to store sensitive information,
 # like your database password, in your source code. If your source code is
@@ -93,10 +87,8 @@ test:
 production:
   primary: &primary_production
     <<: *default
-    database: mie_production
-    username: mie
-    password: <%= ENV['MIE_DATABASE_PASSWORD'] %>
+    database: storage/production.sqlite3
   cache:
     <<: *primary_production
-    database: mie_production_cache
+    database: storage/production_cache.sqlite3
     migrations_paths: db/cache_migrate

(Optional) Move the solid_queue tables to a separate database, isolating them from the primary database.

When using SolidQueue before version 0.8, the tables for SolidQueue are created in the primary database. However, after version 0.8, the default behavior of SolidQueue is to create a separate database for SolidQueue. The tables still work in the primary database, but I think this is the best time to separate the database, as done in version 0.8 and later.

mkdir -p db/queue_migrate
mv db/migrate/*solid_queue* db/queue_migrate

Add the queue database for config/database.yml

diff --git a/config/database.yml b/config/database.yml
index bb7d0ea..8d7d368 100644
--- a/config/database.yml
+++ b/config/database.yml
@@ -29,6 +29,10 @@ development:
     <<: *primary_development
     database: storage/development_cache.sqlite3
     migrations_paths: db/cache_migrate
+  queue:
+    <<: *primary_development
+    database: storage/development_queue.sqlite3
+    migrations_paths: db/queue_migrate

   # The specified database role being used to connect to postgres.
   # To create additional roles in postgres see `$ createuser --help`.
@@ -92,3 +96,7 @@ production:
     <<: *primary_production
     database: storage/production_cache.sqlite3
     migrations_paths: db/cache_migrate
+  queue:
+    <<: *primary_development
+    database: storage/production_queue.sqlite3
+    migrations_paths: db/queue_migrate

Then, modify the environment config files production.rb and development.rb to set the SolidQueue connects_to option to connect to the queue database.

diff --git a/config/environments/development.rb b/config/environments/development.rb
index 439bb20..5f359ac 100644
--- a/config/environments/development.rb
+++ b/config/environments/development.rb
@@ -53,6 +53,7 @@ Rails.application.configure do

   # Highlight code that enqueued background job in logs.
   config.active_job.queue_adapter = :solid_queue
+  config.solid_queue.connects_to = { database: { writing: :queue } }
   config.active_job.verbose_enqueue_logs = true

   # Raises error for missing translations.

diff --git a/config/environments/production.rb b/config/environments/production.rb
index 050eed7..4255cda 100644
--- a/config/environments/production.rb
+++ b/config/environments/production.rb
@@ -53,6 +53,7 @@ Rails.application.configure do

   # Replace the default in-process and non-durable queuing backend for Active Job.
   config.active_job.queue_adapter = :solid_queue
+  config.solid_queue.connects_to = { database: { writing: :queue } }

   # Enable locale fallbacks for I18n (makes lookups for any locale fall back to
   # the I18n.default_locale when a translation cannot be found).

And migrate the queue database.

bin/rails db:migrate:queue

Add a concern to auto-gen UUID PK for models.

In SQLite, you can't use auto-generated UUID primary keys, so add a concern to generate a UUID when models are saved to the database.

module UuidPrimaryKey
  def self.included(klass)
    klass.before_create :generate_id
  end

  def generate_id
    self.id = loop do
      uuid = SecureRandom.uuid
      break uuid unless self.class.exists?(id: uuid)
    end
  end
end

Update your migration files from the first one, replacing PostgreSQL-specific statements with SQLite-compatible ones.

• Remove postgres extensions from the migration files
• Change the PK data type from UUID to string
• Change the column type from jsonb to json
• Delete the db/schema.rb file so it will be regenerated when running a migration.

Set up database.

Run bin/rails db:migrate to regenerate db/schema.rb.

bin/rails db:create
bin/rails db:migrate

Run test.

Run tests, and if some tests fail, check the reason for failure and fix it. If it didn't happen due to differences between PostgreSQL and SQLite, modify the models or controllers accordingly.

bin/rails test

Running 170 tests in parallel using 8 processes
Run options: --seed 30856

# Running:

........F

Failure:
ProfileTest#test_#friend_profiles_retuns_friends_profile_models [test/models/profile_test.rb:68]:
--- expected
+++ actual
@@ -1 +1 @@
-[#<Profile id: "349056977", user_id: "980190962", provider: "github", uid: "1234", name: "test profile 1", avatar_url: "https://example.com/avatar/1", created_at: "2025-02-17 14:07:57.561650000 +0000", updated_at: "2025-02-17 14:07:57.561650000 +0000", introduce: nil>, #<Profile id: "1063789403", user_id: "298486374", provider: "github", uid: "5678", name: "test profile 2", avatar_url: "https://example.com/avatar/2", created_at: "2025-02-17 14:07:57.561650000 +0000", updated_at: "2025-02-17 14:07:57.561650000 +0000", introduce: nil>]
+[#<Profile id: "1063789403", user_id: "298486374", provider: "github", uid: "5678", name: "test profile 2", avatar_url: "https://example.com/avatar/2", created_at: "2025-02-17 14:07:57.561650000 +0000", updated_at: "2025-02-17 14:07:57.561650000 +0000", introduce: nil>, #<Profile id: "349056977", user_id: "980190962", provider: "github", uid: "1234", name: "test profile 1", avatar_url: "https://example.com/avatar/1", created_at: "2025-02-17 14:07:57.561650000 +0000", updated_at: "2025-02-17 14:07:57.561650000 +0000", introduce: nil>]



bin/rails test test/models/profile_test.rb:64

.................................................................................................................................................................

Finished in 2.099298s, 80.9795 runs/s, 168.1514 assertions/s.
170 runs, 353 assertions, 1 failures, 0 errors, 0 skips

In this case, only one test has failed. This test checks an array retrieved via the model's through association and returns the correct array, but the order is different. Fix the test to call the order method on the array, ordering it by uid, before making the assertion.

Restore the database with the modified dump files.

sqlite3 storage/development.sqlite3 < tmp/database_backup.sql

# if you dump SolidQueue data into another dump file.
sqlite3 storage/development_queue.sqlite3 < tmp/database_solid_queue_backup.sql

Remove docker-compose.yml

Clean up containers and remove docker-compose.yml file.

docker-compose down
rm docker-compose.yml