[EPIC] In place migration of oc10 to ocis

[butonic]

In place migration of oc10 to ocis

Milestone issue list: Migration

Scenario

This scenario is based on the one for the data_exporter requirements.

Given an instance at https://demo.owncloud.com where user einstein shared a folder /photos with marie who mounted it as /projects/abc this is the flow that is planned to migrate an instance user by user:

• cover reshares
• cover (re-)shares to groups

0. Prerequisites

• Document how to set up OIDC auth and phoenix for oc10
  • https://github.com/owncloud/openidconnect openidconnect support for core
  • Phoenix integration .... core#34484 Phoenix integration into core
  • we assume the oidc idp/iss is https://idp.owncloud.com

1. Set up reverse proxy

1. make oc10 available at https://oc10.owncloud.com as well
2. Introduce OCIS as proxy for https://demo.owncloud.com

• add proxy middleware for reva

OCIS is used to forward requests for unmigrated users to the oc10 instance/domain.

• it keeps a list of migrated users do decide which requests to forward

Migrated users will be hosted by ocis

2. Migrate groups (optional)

Can be skipped if all groups are maintained in an LDAP server that has been configured for oc10 & ocis

• occ export:groups TODO there is only occ groups:list
• occ import:groups TODO use occ group:add

3. Migrate user by user

1. in oc10 export user file and share metadata: occ export:user einstein, occ export:user marie, occ export:user richard

• will export file and share metadata.
  • basic data format is in place: see Data Format V2 data_exporter#77
  • share recipient has no mount point information, but we need it to reconstruct the shares. tracked in export mount points  data_exporter#113
  • the file metadata contains a reference to the share that allows reconstructing the storage and path to the shared file, eg /projects/abc is a reference to type home user einstein and path /photos
• nothing is changed in either instance, a reexport can happen at any time
• TODO create a milestone in the data exporter for needed changes/fixes for these
  • Make file export optional
  • drop /files prefix for path in files.jsonl
  • export mount points
  • reshares are exported with path null
  • Add Extended Share attributes

2. in ocis reva import marie --user-iss https://idp.owncloud.com --user-sub cb1dd81e-6967-44f7-8239-dffbbe319e92

• imports the user marie with iss https://idp.owncloud.com and sub cb1dd81e-6967-44f7-8239-dffbbe319e92 into ocis
  • create the storage if the user does not exist yet
  • add storage registry entry? for now the static registry will use the same storage provider for all the users.
• shares to migrated users are recreated as internal shares (for the first user, no shares will be migrated)
  • due to path based references we need to have the list of oc10 usernames to ocis sub@iss mappings available to properly rewrite the userid in references
• shares to not migrated users are created as federated shares to the old instance
  • with a user id mapping file we can set the permissions in the storage
  • implement a handler for the federated sharing api
    • needs an auth provider for the federated sharing id
• shares from migrated users are mounted
  • can be determined if the exported reference can be resolved to an existing file/folder, including the userid mapping
• shares from non migrated users are skipped, they will be migrated in the next step
• public shares are recreated with the same token
• import file metadata, WIP PR Feature/import etag and mtime cs3org/reva#299 needs Gateway+metadata cs3org/reva#289

Repeat the import and export for as many users as desired

3. s1: occ migrate:shares marie https://demo.owncloud.com

• shares to marie are converted into federated shares to marie@demo.owncloud.com
  • use a username -> iss & sub mapping file
  • creating federated shares will send a federated sharing request to ocis... or not because oc10 thinks it is still responsible for both domains.
  • needs to be done using sql
  • the command needs an API to create shares in OCIS
    • for shares from/to migrated users, internal shares are set up
    • for shares from/to non migrated users, federated shares are created.

4. s1: occ user:delete marie

• but without actually deleting files?
• mark user as migrated for the proxy

5. migrate trash in a dedicated step?
6. migrate versions in a dedicated step?

migrate data (file content)

• data needs to be moved from old ownCloud data directory into new eos layout
  • no files (files_trashbin, files_versions) folder per user, we need to move files to new user home
    • to import versions into EOS, see below
    • to import trash into EOS, see below
  • owner changes from www-data to the actual user (users must be known to the underlying os, so ACLs can be set up correctly)

proxy

• user by user migration needs to send users to either revad or oc10 apache servers
  • task for the authentication / reverse proxy, likely a header that is set during login
  • can we make reva act as a proxy for the old oc10 so we can intercept the traffic and send users to the right instance
    • would that help with sharing?
    • would this allow sharding instances
• how do we authenticate users
  • phoenix supports oidc and oauth2, but no basic auth
  • revad supports oidc and can use basic auth
  • since we have to control the users anyway, in order to set ACLs for guest accounts we may have to manage our own ldap server that can then be used to provide users for an oidc capable IdP

[labkode]

I leave some notes that could be useful for the import.

Versions can be injected into EOS. EOS will create a directory named .sys.v#.<filename> that will contains the versions of the files. The version name is <timestamp>.<something>.

[gonzalhu@lxplus765 ~]$ ls -rlhita /eos/user/g/gonzalhu/RotaRevolution/ | grep by_alarm                                                                                                                                                                                                                                
72812717739606016 -rw-r--r--. 1 gonzalhu it 7.2K Aug  6 13:49 gni_by_alarm.xlsx
         30621015 drwxr-xr-x. 2 gonzalhu it 4.0K Aug  6 13:49 .sys.v#.gni_by_alarm.xlsx

[gonzalhu@lxplus765 ~]$ ls -rlhita /eos/user/g/gonzalhu/RotaRevolution/.sys.v#.gni_by_alarm.xlsx
total 15K
72812679353335808 -rw-r--r--. 1 gonzalhu it 6.3K Aug  6 13:49 1565092158.102aebf0
         30621015 drwxr-xr-x. 2 gonzalhu it 4.0K Aug  6 13:49 .
         30620954 drwxr-xr-x. 2 gonzalhu it 4.0K Aug 20 09:21 ..


[gonzalhu@lxplus765 ~]$ eos root://x.cern.ch file versions /eos/user/g/gonzalhu/RotaRevolution/gni_by_alarm.xlsx                                                                                                                                                                                              
-rw-r--r--   2 gonzalhu it               6427 Aug  6 13:49 1565092158.102aebf0

Trash-bin can also be injected, but trash-bin lives into a different namespace outside user folder:

[root@x (mgm:master mq:master) ~]$ eos ls -l /eos/home-i01/proc/recycle/uid:123/2019/08/20/0
---------x   2 gonzalhu it           10000000 Aug  7 10:31 #:#eos#:#user#:#g#:#gonzalhu#:#RotaRevolution#:#.sys.a#.v#RotaRev.pptx.217512690500000000000000000000000000.00000000102f2a64
---------x   2 gonzalhu it           10000000 Aug  7 10:38 #:#eos#:#user#:#g#:#gonzalhu#:#RotaRevolution#:#.sys.a#.v#RotaRev.pptx.308768256400000000000000000000000000.00000000102f2fb0
drwxr-sr-x   1 gonzalhu it               6427 Aug  6 15:55 #:#eos#:#user#:#g#:#gonzalhu#:#RotaRevolution#:#.sys.v#.gni_by_alarm_october..xlsx#:#.0000000001d34a60.d
-rw-r--r--   2 gonzalhu it               7753 Aug  6 15:55 #:#eos#:#user#:#g#:#gonzalhu#:#RotaRevolution#:#gni_by_alarm_october..xlsx.00000000102bc61f


[root@x (mgm:master mq:master) ~]$ eos -r gonzalhu it recycle ls 2019/08/20
# pre-configuring default route to /eos/user/r/root/
# -use $EOSHOME variable to override
# Deletion Time            UID      GID      SIZE         TYPE          RESTORE-KEY      RESTORE-PATH                                                    
# ==============================================================================================================================
Tue Aug 20 09:21:09 2019   gonzalhu it       10000000     file          00000000102f2a64 /eos/user/g/gonzalhu/RotaRevolution/.sys.a#.v#RotaRev.pptx.217512690500000000000000000000000000
Tue Aug 20 09:21:19 2019   gonzalhu it       10000000     file          00000000102f2fb0 /eos/user/g/gonzalhu/RotaRevolution/.sys.a#.v#RotaRev.pptx.308768256400000000000000000000000000
Tue Aug 20 09:21:56 2019   gonzalhu it       7753         file          00000000102bc61f /eos/user/g/gonzalhu/RotaRevolution/gni_by_alarm_october..xlsx  
Tue Aug 20 09:21:56 2019   gonzalhu it       6427         recursive-dir 0000000001d34a60 /eos/user/g/gonzalhu/RotaRevolution/.sys.v#.gni_by_alarm_october..xlsx/

The hex number after the extension is the recovery key.

For the ACLs, Andreas is implementing a mechanism where you could create a key for a user and use that as the ACL in EOS, without the need to use local users, but this will bring us to the same situation that all data still is owned by only one account.

For our migration we have L7 proxy, https://github.com/cernbox/cboxredirectd/ that understands the requests and forwards users to one system or to another.

[butonic]

• how do we import versions and trash items via the CS3 api?

[refs]

efforts on https://github.com/owncloud/ocis-migration supercede this issue. See https://owncloud.github.io/ocis/deployment/#migrate-an-existing-owncloud-10

[jnweiger]

Elcin's implementation: https://github.com/esgarov/Daten-Import-in-Ocis