Migration from openedX Native

[kbitr]

Hey, I have a customized openedX Native version up and running on an AWS EC2 instance. Now I want to migrate to the docker version provided.

Any suggestions how to do this in the first place?

[regisb]

Basically, all you need to do is to migrate the data in MySQL and Mongodb. Did you make important changes to your platform? Do you have some experience with docker and server administration?

[kbitr]

Hey @regisb,

some changes were made to the platform which weren't documented, but I see some changes in the git repos of the edx-platform etc.

Server administration yes, docker so-so.

[regisb]

Mysql data migration

1. Dump the data from your existing MySQL edxapp database:

   mysqldump -u root --password --host yourmysqlhost  edxapp > /tmp/dump.sql
   

2. Move this dump to a folder shared with the LMS container and run the LMS container:

   mv /tmp/dump.sql /path/to/openedx-docker/data/lms/
   make lms
   

3. Insert the dumped data in the openedx database:

   mysql --user=root --host=mysql --password openedx < /openedx/data/lms/dump.sql
   

You will need to root password to the database, which is indicated in openedx-docker/config/config.json

Mongodb data migration

Similarly, you have to transfer the cs_comments_service and edxapp mongodb databases to cs_comments_service and openedx, respectively.

1. Dump databases:

   mongodump --host=yourmongodbhost --out=/path/to/openedx-docker/data/mongodb/dump
   

2. Connect to mongodb container:

   docker-compose exec mongodb bash
   

3. Import data:

   mongorestore /data/db/dump
   

4. Launch a mongodb shell and rename the edxapp collection to openedx (as per this stackoverflow answer:

$ mongo
db.copyDatabase("edxapp", "openedx");
use edxapp;
db.dropDatabase();

And you should be fine. Please try out these recommendations and come back with the result of your attempt. It would be great if others could benefit from your experience.

[kbitr]

I am trying and will write a Howto, when I'm finished completely.

• mysql dump seems to be successful
• where is the password for the mongoldb located?
• what is about all the resources like /edxapp/data or themes?

[regisb]

where is the password for the mongoldb located?

Do you mean the password for Mongodb in your current native install? It should be in lms.auth.json, in the DOC_STORE_CONFIG entry.

what is about all the resources like /edxapp/data or themes?

Hmmm that's right, I had forgotten about those. In openedx-docker, uploads are stored in data/lms/uploads. In the native install they are in /edx/var/edxapp/media/. You should only have to copy-paste them.

As for themes... Well I'm not so sure, as themes are not (yet) supported out of the box in openedx-docker. You might want to check this issue.

[regisb]

@kbitr how did it go?

[kbitr]

@regisb I didn't find the time to yet, I will update this when I did. ✌🏻

[kbitr]

@regisb

Okay, I had a few tries and this is how far I got:

mysql:
-mysql exported
-copied it to the mounted docker volume, then:
mysql --user=root --password openedx < /openedx/data/lms/dump.sql
results in
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

mongodb:
mongodump --host=localhost --out=/tmp/data/mongodb/dump
results in
connected to: localhost assertion: 13 not authorized on admin to execute command { getParameter: 1, authSchemaVersion: 1 }
equally:
mongodump --host localhost --out=/tmp/dump -u edxapp -p (pass from lms.auth.json file)
results in
connected to: localhost assertion: 18 { ok: 0.0, errmsg: "auth failed", code: 18 }

I don't know how to continue by now.

[regisb]

The command to interact with the MySQL server from inside the container was incorrect. Here's the correct command:

mysql --user=root --host=mysql --password openedx < /openedx/data/lms/dump.sql

(I'll edit my previous answer)

As for interacting with your current MongoDb server, I don't know how it's configured so it's kind of hard to make a recommentation. You need to find the right password. Maybe I can help you if you send me privately your lms.auth.json.

[kbitr]

mysql:

• mysql --user=root --host=mysql --password openedx < /openedx/data/dump.sql
  The path was /openedx/data/dump.sql and not /lms/ when copied to openedx-docker/data/lms/, the password was asked and the password inside the lms config worked.

mongo:
Turned out to be

• mongodump --host localhost --port 27017 --db edxapp --username edxapp --password "<pass>" --out /tmp/mongodump
• transferred the files
• mongorestore -d edxapp /data/db/dump/ (hope -d openedx was wrong (duplicates))
  Attention: Doing that the whole server froze with all websites etc. on it.

scp and chown were used besides that. I transferred the theme with the newly merged themes folder. So far so good, still my openedx looks the very same as before. (just the demo course from docker). Additionally, where does the content of native's /edx/var/edxapp/data belong? Anything else I have to transfer?

[regisb]

Awesome! You will probably have to move the content of /edx/var/edxapp/data to ./data/lms and/or ./data/cms. It's hard to say without viewing the folder content.

[kbitr]

The folder includes names of users.

Furthermore, an error 500 is apparent.

[regisb]

Please check the logs in order to debug 500 errors: https://github.com/regisb/openedx-docker/#logging

[kbitr]

I am not quite sure what I am looking for there. And do not want to spam this issue. 😅

[regisb]

No worries! Just trigger a 500 error by visiting the lms page mentioned above, and then paste here the output of docker-compose logs --tail=200.

[kbitr]

Here we go, I added timestamps.

log.txt

[regisb]

Here is the relevant error:

OperationalError: (1054, "Unknown column 'social_auth_partial.timestamp' in 'field list'")

This is likely due to a missing migration. Are you sure that the previous install was based on Hawthorn, and that all migrations had been correctly applied?

Please run make lms, and then in the container: ./manage.py lms showmigrations. This will show missing migrations (please paste the output here). You can then run ./manage.py lms migrate in the container (or make migrate-openedx on the host) to apply missing migrations.

[regisb]

@kbitr did you make some progress on this issue?

[kbitr]

@regisb Sorry, have to freeze this for a while. But, you were right, actually it was a Ginkgo release which I wasn't aware of. Have some trouble with an edx service provider who originally installed this and documented nothing. 😑

[regisb]

I feel your pain! Maybe you can try to use the ginkgo branch?
I'll close this now. Feel free to re-open if you start working on this again and you bump into a bug.

[mostafahussein]

@regisb I am not sure if i should go with opening my own issue or add it in here, I have ficus installation and I want to go with hawthorn, but it seems that the mysql schema is different, what should i do to migrate it ?

[regisb]

@mostafahussein Indeed, this is a very different issue than the original one. Please don't post here. Also, Tutor was not created at the time of the Ficus release, so Tutor does not support migrating from Ficus.

In general, migrating from older versions of Open edX depends heavily on your own setup. If you have a very customised setup, with a fork of edx-platform for example, then a lot of work is required and no one will help you for free. If you installed the official Open edX, release, you should follow the standard upgrade path from version to version (Ficus -> Ginkgo -> Hawthorn -> Ironwood).

I noticed you already posted questions in the Open edX slack channels and discussion groups. It is very hard for people to help you if you don't provide more details in your question. In particular, you need to describe what you already tried. I strongly encourage you to read the following very well-written articles:
https://stackoverflow.com/help/how-to-ask
http://www.catb.org/esr/faqs/smart-questions.html

[mostafahussein]

@regisb Thanks, I am not sure which question you mean but yes i posted to discussion group, and i already mentioned that i tried to dump database and then restore it as mentioned in the official way to upgrade which is similar to your comment: #87 (comment) maybe i was not clear enough in the explanation. anyway i will try to upgrade it from version to version so i can use it hawthorn in docker. Thanks again

[misilot]

I put together my journey / steps here in case anyone stumbles upon this issue.