Migration: Import compressed archive file handling

[tclaus]

It fixes the '# TODO: archive is likely to be a .json.gz file' in migration_service.rb .

A profile export is downloaded as a gz file, which can not be imported directly before this PR without unzipping.
By uncompressing a gz file, it also handles a zip file (maxOS and Windows create Zip on compressing files by default)
It won't overwrite any existing files.
It removes the uncompressed file after handling it (without removing plain json file, if uncompressed file was given)
@Flaburgan @cmrd-senya What do you think?

[tclaus]

Will be integrated into bigger PR with UI

[Flaburgan]

The smaller the PRs are, the easier it is to review them. If that works independently of the UI, I'm in favor a separate PR (I can re-open this one). You probably want to add tests to see it merged though ;)

[tclaus]

Thats an argument.
The mentioned "bigger" Branch currently evolves here: https://github.com/tclaus/diaspora/tree/upload_migration_files

(But it will sit on top of this, I need the zip file handling)

[weex]

(But it will sit on top of this, I need the zip file handling)

I read this to mean this is WIP or draft.

[tclaus]

I read this to mean this is WIP or draft.

No, it's not really a WIP or Draft.

This PR is part of the "Migration" Feature. The zipped file support can be used / merged without the other parts.

[Flaburgan]

So, I'm trying this locally with the rake task and it fails:

bin/rake accounts:migration 
WARNING: diaspora.yml is deprecated and will no longer be read in diaspora 0.9.
         Please copy over diaspora.toml.example to diaspora.toml and migrate your settings from diaspora.yml.
/var/lib/gems/2.7.0/gems/faraday-0.15.4/lib/faraday/options.rb:166: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead
/var/lib/gems/2.7.0/gems/faraday-0.15.4/lib/faraday/options.rb:166: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead
/var/lib/gems/2.7.0/gems/faraday-0.15.4/lib/faraday/options.rb:166: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead
/var/lib/gems/2.7.0/gems/faraday-0.15.4/lib/faraday/options.rb:166: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead
/var/lib/gems/2.7.0/gems/faraday-0.15.4/lib/faraday/options.rb:166: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/type.rb:27: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/type/adapter_specific_registry.rb:9: warning: The called method `add_modifier' is defined here
/var/lib/gems/2.7.0/gems/faraday-0.15.4/lib/faraday/rack_builder.rb:55: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead
/var/lib/gems/2.7.0/gems/actionpack-5.2.6/lib/action_dispatch/middleware/stack.rb:37: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/var/lib/gems/2.7.0/gems/actionpack-5.2.6/lib/action_dispatch/middleware/static.rb:111: warning: The called method `initialize' is defined here
WARNING: Sidekiq::Web.sessions= is no longer relevant and will be removed in Sidekiq 7.0. /var/lib/gems/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/web.rb:75:in `set'
Account migration is requested
Enter the archive path: ../path_diaspora_data_l1hBiuYRhMBsbU0TgzZPwQ.json.gzip
Enter the new user name: test
Archive path: ../path_diaspora_data_l1hBiuYRhMBsbU0TgzZPwQ.json.gzip
New username: test
rake aborted!
NoMethodError: undefined method `content_type' for nil:NilClass
/home/fla/diaspora/diaspora/app/services/migration_service.rb:83:in `zip_file?'
/home/fla/diaspora/diaspora/app/services/migration_service.rb:76:in `archive_file'
/home/fla/diaspora/diaspora/app/services/migration_service.rb:72:in `archive_validator'
/home/fla/diaspora/diaspora/app/services/migration_service.rb:13:in `validate'
/home/fla/diaspora/diaspora/lib/tasks/accounts.rake:12:in `block (2 levels) in <top (required)>'
Tasks: TOP => accounts:migration
(See full trace by running task with --trace)

[tclaus]

I see the problem.
But in the meantime, test it with gz or zip file suffix.

[Flaburgan]

Wow you're actually right, it's because the path is wrong, the file is named .gz but I typed .gzip. Well I guess we should have an error message or something saying the file can't be found instead of a stacktrace.

I have tons of warning now but it works:

fla@enigma:~/diaspora/diaspora$ bin/rake accounts:migration 
WARNING: diaspora.yml is deprecated and will no longer be read in diaspora 0.9.
         Please copy over diaspora.toml.example to diaspora.toml and migrate your settings from diaspora.yml.
/var/lib/gems/2.7.0/gems/faraday-0.15.4/lib/faraday/options.rb:166: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead
/var/lib/gems/2.7.0/gems/faraday-0.15.4/lib/faraday/options.rb:166: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead
/var/lib/gems/2.7.0/gems/faraday-0.15.4/lib/faraday/options.rb:166: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead
/var/lib/gems/2.7.0/gems/faraday-0.15.4/lib/faraday/options.rb:166: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead
/var/lib/gems/2.7.0/gems/faraday-0.15.4/lib/faraday/options.rb:166: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/type.rb:27: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/type/adapter_specific_registry.rb:9: warning: The called method `add_modifier' is defined here
/var/lib/gems/2.7.0/gems/faraday-0.15.4/lib/faraday/rack_builder.rb:55: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead
/var/lib/gems/2.7.0/gems/actionpack-5.2.6/lib/action_dispatch/middleware/stack.rb:37: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/var/lib/gems/2.7.0/gems/actionpack-5.2.6/lib/action_dispatch/middleware/static.rb:111: warning: The called method `initialize' is defined here
WARNING: Sidekiq::Web.sessions= is no longer relevant and will be removed in Sidekiq 7.0. /var/lib/gems/2.7.0/gems/sidekiq-6.2.1/lib/sidekiq/web.rb:75:in `set'
Account migration is requested
Enter the archive path: ../test_diaspora_data_l1hBiuYRhMBsbU0TgzZPwQ.json.gz
Enter the new user name: testmigration
Archive path: ../test_diaspora_data_l1hBiuYRhMBsbU0TgzZPwQ.json.gz
New username: testmigration
/var/lib/gems/2.7.0/gems/json-schema-2.8.1/lib/json-schema/schema/reader.rb:121: warning: calling URI.open via Kernel#open is deprecated, call URI.open directly or use URI#open
/var/lib/gems/2.7.0/gems/activemodel-5.2.6/lib/active_model/type/integer.rb:13: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/var/lib/gems/2.7.0/gems/activemodel-5.2.6/lib/active_model/type/value.rb:8: warning: The called method `initialize' is defined here
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/connection_adapters/postgresql/oid/specialized_string.rb:12: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/var/lib/gems/2.7.0/gems/activemodel-5.2.6/lib/active_model/type/value.rb:8: warning: The called method `initialize' is defined here
/var/lib/gems/2.7.0/gems/faraday-0.15.4/lib/faraday/rack_builder.rb:55: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/transactions.rb:212: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/connection_adapters/abstract/database_statements.rb:260: warning: The called method `transaction' is defined here
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/connection_adapters/abstract/transaction.rb:171: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/connection_adapters/abstract/transaction.rb:97: warning: The called method `initialize' is defined here
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/relation/delegation.rb:132: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/relation.rb:25: warning: The called method `initialize' is defined here
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/persistence.rb:705: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/var/lib/gems/2.7.0/gems/activerecord-5.2.6/lib/active_record/timestamp.rb:105: warning: The called method `_update_record' is defined here

[tclaus]

My workflow was to drag and drop the file from finder (Mac) directly into the terminal. The Filepath is then copied.
But it's an issue that needs to get fixed.

[SuperTux88]

You can just write ensure in the MigrationService in perform! (above the line of calling remove_intermediate_file there), that way remove_intermediate_file doesn't need to be called twice (and it also ensures that the temp files get cleaned up when perform! is called from somewhere else). It also means remove_intermediate_file can be private since it doesn't need to be called from external.

[tclaus]

@SuperTux88 Thanks for reviewing this. In context of #8274 this rake file was more refactored to eliminate duplicate code. So I think for now this file can be as it is.

[SuperTux88]

This was more a request to also change MigrationService.perform! and add the ensure directly there (and when remove_intermediate_file doesn't need to be called here anymore it can also be made private).

  def perform!
    find_or_create_user
    import_archive
    run_migration
  ensure
    remove_intermediate_file
  end

[SuperTux88]

Squashed and merged, thanks 🍪