feat: User migration scripts #401

jrconlin · 2020-01-07T20:08:24Z

Issue: #286

Description

Several scripts to assist in user migration from MySQL to spanner. Consult the README.md file for details on use and configuration.

Testing

Set up a local syncserver, connect, and store some BSO data.
Configure the move_dsns.lst config file to point to the correct mysql and spanner instances. (e.g.

mysql://test:test@localhost/syncstorage
spanner://projects/sync-spanner-dev-225401/instances/spanner-test/databases/sync_schema3

Store the known local syncserver userID into move_users.lst (e.g. 1)
After running setup (python3 -m venv venv && . venv/bin/activate && pip install -r requirements.txt) Run the python3 migrate_user.py script.
user data record should be copied from mysql to spanner, Note the
Processsing... # -> ###:###
which displays the original mysql userid and the spanner fxa_uid:fxa_kid

Issue(s)

Issue #286, mozilla-services/services-engineering#18

Several scripts to assist in user migration from MySQL to spanner. Issue: #286

tublitzed · 2020-01-07T20:23:05Z

@jrconlin just to clarify, this is related to #18, not 186 or 286, y?

jrconlin · 2020-01-07T22:35:25Z

Sigh, typo. I've updated to address the two issues I believe are related. (Hilariously, I got the issue
correct in the git PR.)

…nto user_migration

tublitzed · 2020-01-08T16:22:50Z

np, just making sure I wasn't missing something

tools/user_migration/migrate_user.py

tools/README.md

tools/user_migration/migrate_user.py

There still be dragons

…age-rs into user_migration

tublitzed · 2020-01-13T19:40:29Z

@jrconlin following the setup instructions here, regarding step 2: where does the move_dsns.lst file live? I'm not seeing it here in this repo...

tublitzed · 2020-01-13T19:48:08Z

^ ah, nvm. I see now. Following the example here going to assume I create this file and that it's used as the dsns arg. Trying that now....

* switch to `logging` for logging * use spanner `collections` table as one source of truth about collection ID numbers * use spanner.transaction * gracefully fail on user previously migrated. * remove some dragons

tublitzed · 2020-01-14T15:09:04Z

When I run python3 migrate_user.py --dsns move_dsns.lst --users move_users.lst (with the .lst files in the same dir as the script) I see the following error:

  File "migrate_user.py", line 380, in <module>
    main()
  File "migrate_user.py", line 365, in main
    databases[dsn.scheme] = conf_db(dsn)
  File "migrate_user.py", line 135, in conf_db
    return conf_mysql(dsn)
  File "migrate_user.py", line 112, in conf_mysql
    database=dsn.path[1:]
  File "/Users/tublitzed/GIT/syncstorage-rs/tools/user_migration/venv/lib/python3.7/site-packages/mysql/connector/__init__.py", line 179, in connect
    return MySQLConnection(*args, **kwargs)
  File "/Users/tublitzed/GIT/syncstorage-rs/tools/user_migration/venv/lib/python3.7/site-packages/mysql/connector/connection.py", line 95, in __init__
    self.connect(**kwargs)
  File "/Users/tublitzed/GIT/syncstorage-rs/tools/user_migration/venv/lib/python3.7/site-packages/mysql/connector/abstracts.py", line 716, in connect
    self._open_connection()
  File "/Users/tublitzed/GIT/syncstorage-rs/tools/user_migration/venv/lib/python3.7/site-packages/mysql/connector/connection.py", line 210, in _open_connection
    self._ssl)
  File "/Users/tublitzed/GIT/syncstorage-rs/tools/user_migration/venv/lib/python3.7/site-packages/mysql/connector/connection.py", line 142, in _do_auth
    auth_plugin=self._auth_plugin)
  File "/Users/tublitzed/GIT/syncstorage-rs/tools/user_migration/venv/lib/python3.7/site-packages/mysql/connector/protocol.py", line 102, in make_auth
    auth_data, ssl_enabled)
  File "/Users/tublitzed/GIT/syncstorage-rs/tools/user_migration/venv/lib/python3.7/site-packages/mysql/connector/protocol.py", line 58, in _auth_response
    auth = get_auth_plugin(auth_plugin)(
  File "/Users/tublitzed/GIT/syncstorage-rs/tools/user_migration/venv/lib/python3.7/site-packages/mysql/connector/authentication.py", line 191, in get_auth_plugin
    "Authentication plugin '{0}' is not supported".format(plugin_name))
mysql.connector.errors.NotSupportedError: Authentication plugin 'caching_sha2_password' is not supported
(venv)

Anyone else seeing this? I'm running from OSX, MySQL 8.0.18 if that helps.

Edit: I also kept poking around and tried with GOOGLE_APPLICATION_CREDENTIALS set correctly too if that helps. Not sure if the mysql-connector needs some extra configuration in order to get caching_sha2_password working...

tublitzed · 2020-01-14T15:31:26Z

Looks like this might be a known issue with the caching_sha2_password plugin against newer MySQL versions

I have tried the suggested alter workaround to no avail. @jrconlin -what version of MySQL are you testing this against? Looks like 8.x might just be too new for caching_sha2_password...in which case I can try against an older version. Figured I'd check though before going through that :)

(FWIW, I don't think it matters if this script might require an older MySQL version, just noting this for anyone else trying to test this out)

pjenvey

we can merge as is but this needs a couple more bits:

collection_id creation in transaction
(probably) handle users exceeding batch limit
how to handle errors during a user migration. can the entire user's move be in one transaction (so if an error happens, we cleanly abort)? and if we abort a user's migration should we retry? or just report the user id that failed for fixing up later?

tools/user_migration/migrate_user.py

…nto user_migration

jrconlin · 2020-01-14T23:51:59Z

@tublitzed My server is reporting it's "5.7.28-0ubuntu0.18.04.4", which is what I get out of apt/debian, which might be stable locked for ubuntu.

…nto user_migration

jrconlin · 2020-01-14T23:55:33Z

@pjenvey Fixed the collection_id to be in the same transaction, but tempted to leave the other two points as future bugs, mostly because i'll need to create a few datasets to try and replicate those so I understand what the breaking points are.

tools/user_migration/migrate_user.py

* pass transaction handle to collection_id generator * reserve collection_ids < 100

tublitzed · 2020-01-16T15:50:19Z

@jrconlin I downgraded to MySQL 5.7, still seem same errors from OSX. It'd probably be good to confirm we can get at least a few people able to run this locally so we can effectively test client behaviour internally before handing off to QA. IIRC the plan was to test locally first, so I don't want that to fall entirely to you if you're the only one who can run this :)

Also curious if maybe it's just my machine. @pjenvey are you able to run this locally?

jrconlin · 2020-01-16T15:54:37Z

Also kinda curious if this might be a mac thing vs. linux thing. (if it is, i am even LESS enamored with mysql).

IIUC, the scripts would be run from debian sh/bash shells, since that's what GCP provides. I'm running ubuntu, which is a debian fork, so the libraries should match up reasonably well.

tools/user_migration/migrate_user.py

pjenvey · 2020-01-23T18:30:55Z

tools/user_migration/migrate_user.py

+        cursor = database.cursor()
+        cursor.execute(
+            """create table if not exists migration (
+                fxa_uid varchar(255) NOT NULL PRIMARY KEY,


With this being for the python syncstorage side to reference, I think it will be much easier to use its "legacy" userid vs syncstorage-rs's fxa_uid

Yeah, but "userid" is kinda db dependent since it can come off of the db row. fxa_uid isn't. I'd rather there be zero chance of collision here, so I'd prefer to use fxa_uid.

Cool, I forgot it's really not difficult to get at either (it's already setup on the user dict/object "fxa_uid" field).

tools/user_migration/migrate_user.py

…age-rs into user_migration

feat: User migration scripts

3500b9b

Several scripts to assist in user migration from MySQL to spanner. Issue: #286

jrconlin requested a review from a team January 7, 2020 20:08

Merge branch 'master' of github.com:mozilla-services/syncstorage-rs i…

cf6a607

…nto user_migration

Merge branch 'master' into user_migration

b7bce30

jrconlin mentioned this pull request Jan 9, 2020

User Migration Meta bug #407

Closed

pjenvey requested changes Jan 10, 2020

View reviewed changes

jrconlin added 2 commits January 10, 2020 15:53

f r's

edcceb9

There still be dragons

Merge branch 'user_migration' of github.com:mozilla-services/syncstor…

a3ebe76

…age-rs into user_migration

f r's

10a854e

* switch to `logging` for logging * use spanner `collections` table as one source of truth about collection ID numbers * use spanner.transaction * gracefully fail on user previously migrated. * remove some dragons

jrconlin requested a review from pjenvey January 14, 2020 00:01

pjenvey previously approved these changes Jan 14, 2020

View reviewed changes

tools/user_migration/migrate_user.py Outdated Show resolved Hide resolved

tools/user_migration/migrate_user.py Outdated Show resolved Hide resolved

Merge branch 'master' of github.com:mozilla-services/syncstorage-rs i…

a908951

…nto user_migration

jrconlin added 2 commits January 14, 2020 15:52

f r -move collection_id create into transaction

eea51ea

Merge branch 'master' of github.com:mozilla-services/syncstorage-rs i…

32a20f7

…nto user_migration

jrconlin dismissed pjenvey’s stale review via 32a20f7 January 14, 2020 23:53

pjenvey reviewed Jan 15, 2020

View reviewed changes

tools/user_migration/migrate_user.py Outdated Show resolved Hide resolved

pjenvey reviewed Jan 15, 2020

View reviewed changes

tools/user_migration/migrate_user.py Outdated Show resolved Hide resolved

tublitzed mentioned this pull request Jan 15, 2020

Refine migration script mozilla-services/services-engineering#31

Closed

f r's

72ac2cd

* pass transaction handle to collection_id generator * reserve collection_ids < 100

jrconlin requested a review from pjenvey January 16, 2020 17:36

pjenvey reviewed Jan 22, 2020

View reviewed changes

tools/user_migration/migrate_user.py Outdated Show resolved Hide resolved

f get id from token or mysql in that order

59ae411

pjenvey previously approved these changes Jan 23, 2020

View reviewed changes

f mark migration table

c9514ab

jrconlin dismissed pjenvey’s stale review via c9514ab January 23, 2020 01:05

Merge branch 'master' into user_migration

f31a5b4

pjenvey reviewed Jan 23, 2020

View reviewed changes

jrconlin added 2 commits January 23, 2020 13:31

f r's

9d98c60

Merge branch 'user_migration' of github.com:mozilla-services/syncstor…

e979035

…age-rs into user_migration

pjenvey approved these changes Jan 24, 2020

View reviewed changes

jrconlin merged commit cf31fde into master Jan 24, 2020

jrconlin mentioned this pull request Feb 12, 2020

bug: convert migration state to smallint #429

Merged

jrconlin mentioned this pull request Apr 10, 2020

Latest ops requests for migrate_node #587

Merged

jrconlin mentioned this pull request Apr 21, 2020

User migration5 #601

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: User migration scripts #401

feat: User migration scripts #401

jrconlin commented Jan 7, 2020 •

edited

Loading

tublitzed commented Jan 7, 2020

jrconlin commented Jan 7, 2020

tublitzed commented Jan 8, 2020

tublitzed commented Jan 13, 2020

tublitzed commented Jan 13, 2020

tublitzed commented Jan 14, 2020 •

edited

Loading

tublitzed commented Jan 14, 2020

pjenvey left a comment

jrconlin commented Jan 14, 2020

jrconlin commented Jan 14, 2020

tublitzed commented Jan 16, 2020

jrconlin commented Jan 16, 2020

pjenvey Jan 23, 2020

jrconlin Jan 23, 2020

pjenvey Jan 24, 2020

feat: User migration scripts #401

feat: User migration scripts #401

Conversation

jrconlin commented Jan 7, 2020 • edited Loading

Description

Testing

Issue(s)

tublitzed commented Jan 7, 2020

jrconlin commented Jan 7, 2020

tublitzed commented Jan 8, 2020

tublitzed commented Jan 13, 2020

tublitzed commented Jan 13, 2020

tublitzed commented Jan 14, 2020 • edited Loading

tublitzed commented Jan 14, 2020

pjenvey left a comment

Choose a reason for hiding this comment

jrconlin commented Jan 14, 2020

jrconlin commented Jan 14, 2020

tublitzed commented Jan 16, 2020

jrconlin commented Jan 16, 2020

pjenvey Jan 23, 2020

Choose a reason for hiding this comment

jrconlin Jan 23, 2020

Choose a reason for hiding this comment

pjenvey Jan 24, 2020

Choose a reason for hiding this comment

jrconlin commented Jan 7, 2020 •

edited

Loading

tublitzed commented Jan 14, 2020 •

edited

Loading