Skip to content

Add data dumps#2047

Open
ArtOfCode- wants to merge 27 commits into
developfrom
art/data-dumps
Open

Add data dumps#2047
ArtOfCode- wants to merge 27 commits into
developfrom
art/data-dumps

Conversation

@ArtOfCode-
Copy link
Copy Markdown
Member

@ArtOfCode- ArtOfCode- commented May 12, 2026

Add data dumps. Weekly export of the entire database minus anything sensitive, uploaded to S3 and made available via a new page linked in the footer. Also adds an option for manually-created dump records, intended for quarterly uploads to Archive.org.

Incorporates #1950 by cherry-pick.

@ArtOfCode- ArtOfCode- requested review from Oaphi and cellio May 14, 2026 10:07
@ArtOfCode- ArtOfCode- changed the title Add some more data dump skeleton Add data dumps May 14, 2026
@ArtOfCode- ArtOfCode- changed the base branch from 0valt/1918/data-dump to develop May 14, 2026 10:10
@ArtOfCode- ArtOfCode- linked an issue May 14, 2026 that may be closed by this pull request
@codecov
Copy link
Copy Markdown

codecov Bot commented May 14, 2026

Codecov Report

❌ Patch coverage is 86.44068% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.90%. Comparing base (9c10754) to head (f64ea30).

Files with missing lines Patch % Lines
app/jobs/data_dump_job.rb 88.88% 5 Missing ⚠️
app/controllers/dumps_controller.rb 60.00% 2 Missing ⚠️
app/models/dump.rb 87.50% 1 Missing ⚠️
Additional details and impacted files
Components Coverage Δ
controllers 76.02% <60.00%> (-0.06%) ⬇️
helpers 85.32% <100.00%> (+0.01%) ⬆️
jobs 71.85% <88.88%> (+4.97%) ⬆️
models 93.02% <87.50%> (-0.03%) ⬇️
tasks 61.11% <ø> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@cellio
Copy link
Copy Markdown
Member

cellio commented May 15, 2026

After switching to the branch and running the migrations, and before running any job to generate dumps, I hit the "data dumps" link in the footer and got:

ActionView::Template::Error (undefined method `title' for nil):

Causes:
NoMethodError (undefined method `title' for nil)
    29:   </thead>
    30:   <tbody>
    31:     <tr>
    32:       <td><%= @latest.title %></td>
    33:       <td><%= @latest.file.filename %></td>
    34:       <td><%= @latest.created_at.strftime('%Y-%m-%d') %></td>
    35:       <td><%= link_to 'Download', rails_blob_path(@latest.file, disposition: 'attachment') %></td>
  
app/views/dumps/index.html.erb:32

Let's defend against that; even if we intend to generate a dump immediately after deploying, (a) there might be a delay and (b) we can't assume other instances will do so.

@cellio
Copy link
Copy Markdown
Member

cellio commented May 15, 2026

Running the job from the Rails console produced an access error -- looks like we're missing a grant for perform.

:001> DataDumpJob.perform_later
Performing DataDumpJob (Job ID: 173732ce-812a-4dae-bd88-b6e324228c46) from Inline(default) enqueued at 2026-05-15T03:01:18.998364424Z
Error performing DataDumpJob (Job ID: 173732ce-812a-4dae-bd88-b6e324228c46) from Inline(default) in 17.49ms: ActiveRecord::ConnectionFailed (Mysql2::Error::ConnectionError: Access denied for user 'qpixel'@'%' to database 'qpixel_dump'):

(Stack trace follows; I assume you don't need all that.)

@ArtOfCode-
Copy link
Copy Markdown
Member Author

Running the job from the Rails console produced an access error -- looks like we're missing a grant for perform.

I deliberately didn't include this as part of the job, as the DB user running the export shouldn't have GRANT permissions. I did it manually - I'll add some error handling for that specifically that puts a warning in the job log.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a system for regular and/or on-demand data dumps

3 participants