import: Fix rendered_content in imported messages.

[rheaparekh]

After the messages have been imported, set the rendered_content of the messages instead of
leaving its value to be None.

Fixes #9168

[zulipbot]

Hello @zulip/server-misc members, this pull request was labeled with the "area: export/import" label, so you may want to check it out!

[timabbott]

I think we want something like this. The reason is that in production use, it's quite annoying to have the import crash here (because the tool isn't designed to be rerun without starting from the beginning), whereas it's only mildly annoying to have a couple messages that didn't markdown-render at first.

 
         re_map_foreign_keys(data, 'zerver_message', 'id', related_table='message', id_field=True)
         bulk_import_model(data, Message)
-        fix_message_rendered_content(data, 'zerver_message')
+        try:
+            fix_message_rendered_content(data, 'zerver_message')
+        except Exception as e:
+            logging.warning("Error in markdown rendering for message ID %s; continuing" % (<message_id>))

Needs some cleanup. And then I think we can have a tool that tries to render any un-rendered messages that one can run afterwards.

[rheaparekh]

@timabbott I have updated this

[timabbott]

I tweaked the logging code to print something to show where the time goes for markdown-rendering, and it's definitely in total a huge fraction of the import time:

2018-08-09 22:02:46.447 INFO [] Importing message dump /tmp/slack-export/messages-000001.json
2018-08-09 22:02:46.694 INFO [] Successfully imported <class 'zerver.models.Message'> from zerver_message.
2018-08-09 22:03:02.790 INFO [] Successfully rendered markdown for messages in dump /tmp/slack-export/messages-000001.json
2018-08-09 22:03:03.225 INFO [] Successfully imported <class 'zerver.models.UserMessage'> from zerver_usermessage.
2018-08-09 22:03:03.235 INFO [] Importing message dump /tmp/slack-export/messages-000002.json
2018-08-09 22:03:03.489 INFO [] Successfully imported <class 'zerver.models.Message'> from zerver_message.
2018-08-09 22:03:19.966 INFO [] Successfully rendered markdown for messages in dump /tmp/slack-export/messages-000002.json
2018-08-09 22:03:20.529 INFO [] Successfully imported <class 'zerver.models.UserMessage'> from zerver_usermessage.
2018-08-09 22:03:20.538 INFO [] Importing message dump /tmp/slack-export/messages-000003.json

I'm going to merge this, because correctness is more important than speed, but we should probably look at some combination of (1) optimizing it (I'd be willing to bet that we re-query a lot of data unnecessarily with the import process) and/or (2) doing the markdown rendering in a separate parallel phase after we finish importing all the messages. May be worth starting with (1), though; I know bugdown does a few database queries in its outer layer to get e.g. the set of users in the organization, and we're clearly doing those in a loop here.

[timabbott]

Merged as 2630011.

After doing some code reading, the performance piece looks kinda annoying to optimize. I'll open an issue, which we can tackle a bit later.

[timabbott]

(Huge thanks for fixing this!)

[timabbott]

See #10264 for the performance follow-up issue (probably a lower-priority than some other projects for now, just because it's likely difficult). While thinking about this, I added 4c4b6d1.

[rheaparekh]

Thank you for the review!