Strip file_error and rescue unknown attributes in raw data archive import - #3564
Merged
Freika merged 1 commit intoSep 10, 2026
Conversation
Freika
deleted the
detail/bug-fix/strip-file-error-and-rescue-unknown-attributes-in-a3921c
branch
September 10, 2026 20:16
pull Bot
pushed a commit
to AmirulAndalib/dawarich
that referenced
this pull request
Sep 13, 2026
Follow-up to Freika#3564, which merged before these review fixes landed. A file_error row became an archive record with no file attached. The verifier then reports it as unattached, and find_existing_archive treats the slot as taken, so a later export that does carry the file can never fill it. Those rows are now skipped with a warning naming the year, month and chunk. Attributes are sliced to the model's own columns, as the points importer already does, so a column added on the exporting instance is ignored per key rather than dropping every archive row through the rescue.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Detail bug report: View on Detail
Bug
Users::ImportData::RawDataArchivesaborted the entire user-data restore when a singleraw_data_archives.jsonlrow carried afile_errorkey.Users::ExportData#add_archive_file_datawrites afile_errorkey into a row whenever the archive's blob can't be downloaded/decrypted/written (missing blob, rotated encryption key, S3 outage, disk full, etc.).Points::RawDataArchivehas nofile_errorcolumn, so mass-assigning it raisedActiveModel::UnknownAttributeError.ActiveRecord::RecordInvalid, so the exception escaped toUsers::ImportData#import's top-levelActiveRecord::Base.transaction, rolling back every entity imported earlier inV2Handler#process(settings, areas, places, tags, taggings, imports, exports, trips, stats, digests, notifications, visits, tracks, points). The user saw a generic"unknown attribute 'file_error' for Points::RawDataArchive"failure notification with no indication that one bad raw_data_archive row was the cause.The sibling importers (
imports.rb,exports.rb) already handle this exact case by strippingfile_errorand rescuingActiveModel::UnknownAttributeError;raw_data_archives.rbwas the only one missing both.Fix
In
app/services/users/import_data/raw_data_archives.rb, mirror the sibling importers:create_archive_record: added'file_error'to thearchive_data.except(...)list so the exporter's error marker is not mass-assigned.callrescue clause: broadenedrescue ActiveRecord::RecordInvalidtorescue ActiveRecord::RecordInvalid, ActiveModel::UnknownAttributeErrorso an unknown attribute from a future/desynced exporter skips just that row instead of aborting the whole import.Testing
Added 4 specs to
spec/services/users/import_data/raw_data_archives_spec.rbmirroring the existingfile_errorpattern inexports_spec.rb:file_errorrow still creates the archive (with the key stripped) and restores no file.file_errorrow shaped like real production output (nofile_name) doesn't raise.file_errorrow, valid row) imports all rows without aborting.Verification:
ActiveModel::UnknownAttributeError: unknown attribute 'file_error' for Points::RawDataArchive.(bogus_columnfor the rescue-clause test); re-applied the fix and confirmed all 10 pass.spec/services/users/import_data/— 256 examples, 0 failures.spec/services/users/export_data_spec.rb— 23 examples, 0 failures (thefile_erroremission path at lines 173-214 still produces the input the importer must now tolerate).spec/services/users/export_import_integration_spec.rb— 8 examples, 0 failures.spec/services/users/import_data/v1_handler_spec.rb— 17 examples, 0 failures.bundle exec rails zeitwerk:checkclean;bundle exec rubocopon both changed files reports no offenses.End-to-end V2 round-trip with a real
file_errorrow (export a user whose raw_data_archive blob was deleted from storage, then import into a fresh user viaUsers::ImportData#import) was attempted via a Rails runner script against the test database, but blocked by FactoryBot/RSpec-mock setup not being available outside the RSpec context in the runner; the same scenario is covered by the unit specs above combined with the existing exporter spec that confirmsfile_errorrows are emitted, so the round-trip behavior is verified at the component level.Automatic Fixes PRs can be configured here.