Zulip - RocketChat - Data Converter #20108
Labels
Comments
|
@talheim-it that error message is for an out-of-memory failure. If you are running the tool on a Zullip server with minimal memory, I'd recommend shutting down the Zulip server with It's also possible that either you have a very large Rocket.Chat dataset or there's a bug with the conversation tool that makes it consume more memory than it should; we should investigate those possibilities if shutting down the server while you run the conversion doesn't work. |
|
Hello @zulip/server-misc members, this issue was labeled with the "area: export/import" label, so you may want to check it out! |
|
Is this specific to the Rocket.Chat data conversion? IIRC the Slack version scales nicely to large data. |
|
@timabbott I am testing it today. Thanks for the info |
|
@timabbott I run into the next issue after I increased the memory to 16GB of the vm: Seems like an issue with the channel conversion? Btw you can ignore the path. I set a symlink from /home/zulip to /opt/zulip to install it there since I want to exchange the existent rocket.chat server with it. |
|
Added additional informations about the server to the issue. |
|
@rht I only tested it with Rocket.Chat. I takes a lot memory. I see it permanently Growing. Before all was fine with 4 CPUs, 8GB Memory, 4GB Swap. Now I had to double the Memory to 16GB to get the next error. |
|
@talheim-it with your new exception, can you |
|
I added it on line 152, before the error occurs in the rocketchat.py: This is the output: |
|
@timabbott anything else I can do here? I am stucked within the migration. |
|
@garg3133 FYI as the original author of the tool; can you help debug? |
|
I haven't read this format, but I think what's going on is that Rocket.Chat's Omnichannel/livechat channels use the
? It's likely that a better choice for how to structure the import would be to put these all in a single |
|
Yes, Rocket.Chat uses the I've opened #20153, which should fix this.
Yes, that's what I was thinking. |
|
@garg3133 thanks, I have another issue now: Background: The user is not existing anymore as user, so it seems that the error occurs because the reaction is still in the database, but the user not. Any option to handle this? "username" at KeyError is a placeholder for the username. |
|
@talheim-it You can try changing the line to: rc_user_id = username_to_user_id_map.get(username)And then add the following below it: if not rc_user_id:
continueSo, the complete block of code will be: for username in usernames:
rc_user_id = username_to_user_id_map.get(username)
if not rc_user_id:
continue
user_id = user_id_mapper.get(rc_user_id)
reactions_list.append({"name": name, "user_id": user_id}) |
|
after applying all patches mentioned here, even latest one from @garg3133
|
|
@staffcop did you install the latest Version from git plus added the last change? |
yes I had to increase memory to 16gb, upgrade to main git version, and now I am fiddling inside rocketchat.py |
|
@staffcop Can you put a |
|
I updated it as above: Now it throws following info: I am trying to debug it. |
|
@talheim-it Actually it's |
|
@garg3133 , you are absolutely right, my fault :) Just a new issue: |
|
@garg3133 I try to understand the KeyError. Is it related to the user_id? |
|
I was able to destroy database in the process with no backup, tomorrow
…Sent from my iPhone
On 4 Nov 2021, at 22:42, Priyansh Garg ***@***.***> wrote:
@talheim-it Actually it's username_to_user_id_map.get(username) and not username_to_user_id_map.get[username] :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
|
|
@garg3133 if I can do anything here or send you the dump to check please let me know :) |
|
@talheim-it This error must also be due to the deleted user you told earlier about. Can you try putting the line 447 user = user_handler.get_user(user_id=user_id)inside try-except block? The code would be something like this: for user_id in mention_user_ids:
try:
user = user_handler.get_user(user_id=user_id)
except KeyError:
continue
rc_mention = "@{short_name}".format(**user)
zulip_mention = "@**{full_name}**".format(**user)
content = content.replace(rc_mention, zulip_mention) |
|
@garg3133 seems it worked but hanging at another key: |
|
Try replacing upload = dict(
path=s3_path,
realm_id=realm_id,
content_type=upload["type"],
user_profile_id=user_id,
last_modified=fileinfo["created"],
user_profile_email=user_handler.get_user(user_id=user_id)["email"],
s3_path=s3_path,
size=fileinfo["size"],
)with try:
user_profile_email = user_handler.get_user(user_id=user_id)["email"]
except KeyError:
user_profile_email = "dummyemail@email.com" # or any other dummy email
upload = dict(
path=s3_path,
realm_id=realm_id,
content_type=upload["type"],
user_profile_id=user_id,
last_modified=fileinfo["created"],
user_profile_email=user_profile_email,
s3_path=s3_path,
size=fileinfo["size"],
) |
|
@garg3133 did: I owe you something for your work on this, I am open to fund it with 100,00€ as support. |
|
@talheim-it Can you put a Thanks a lot for your offer, but I'm not really sure if I'm allowed to take it. I'd have to consult someone from the Zulip core team. |
|
@garg3133 there we go: This is the last message: I am writing @timabbott about it. |
|
@talheim-it Okay, so it's a message with just an attachment. In that case, replacing |
|
@garg3133 looks good. The migration runs now. Just need to start importing. |
|
@garg3133 now when I run the import script: |
|
@talheim-it For me, a similar kind of error occurs when I try to import more than once from the same exported files. Can you try to delete the current export and run the migration again and then try to import the organization? (As you've already run it for the |
|
@garg3133 I am checking it an come back to you in the next 30 minutes. |
|
@garg3133 I tried to reinstall zulip and updated it to the latest git version which was chat.zulip.org branch. What I get now is: Maybe that was added few days ago? |
|
No such change was added a few days ago. It seems that Can you |
|
@garg3133 you are right: I added it above: |
|
In that case, you can remove the last argument from this line, making it as: realm = make_realm(realm_id, realm_subdomain, domain_name)and replace the from django.utils.timezone import now as timezone_now
def make_realm(
realm_id: int, realm_subdomain: str, domain_name: str
) -> ZerverFieldsT:
created_at = float(timezone_now().timestamp())
zerver_realm = build_zerver_realm(realm_id, realm_subdomain, created_at, "Rocket.Chat")
realm = build_realm(zerver_realm, realm_id, domain_name)
# We may override these later.
realm["zerver_defaultstream"] = []
return realmBut I don't understand where did the data go when it was present before? (The data in |
|
@garg3133 I made a new dump and then it happened when I downloaded the chat.zulip.org branch for v5 dev. I didn't check the bson files further. |
|
@garg3133 okay the data_migration ran without issues right now. Issues stays the same: The problem is that we talk about livechannels. It seems that because of a live channel I ran into this issue: I am not really sure if it makes sense to keep the omnichannel/livechannel chats or just drop it out by migration since zulip is not having something like this. From the realm.json: I have one user very often in it, can that be the reason why it is not working? |
|
If I see it right it is because the names of the LiveChat user are the ids of the user and so we produce duplications. |
|
Yes, you are right. This error seems to be caused by the duplication of LiveChat channel names, which when converted to Zulip streams create duplicate stream names (no two streams in Zulip can have the same name). As a temporary solution, you can edit the realm.json file and change the names of duplicate streams. It should get imported correctly after that. Ideally, all the livechat channels should be converted to topics in a single Zulip stream, but that is yet to be developed in this tool. |
|
@garg3133 I would prefer to drop the livechat since it is completely useless to import the communication into Zulip. I see no benefit on it since it is for communication with customers from website etc. I sit over an hour to fix duplications, that is a real waste of time if there is no clear reason to import it. Anyway I am running into the next issue by importing: When I look into the realm.json file I see no user with the id 80 in zerver_userprofile |
garg3133
added a commit
to garg3133/zulip
that referenced
this issue
Nov 7, 2021
This resolves the issues reported in zulip#20108, major chunk of which were due to the incomplete support for importing the livechat streams/messages in the tool. So, it's best not to import any livechat streams/messages for now until a complete support for importing the same is developed.
timabbott
pushed a commit
that referenced
this issue
Nov 7, 2021
This resolves the issues reported in #20108, major chunk of which were due to the incomplete support for importing the livechat streams/messages in the tool. So, it's best not to import any livechat streams/messages for now until a complete support for importing the same is developed.
garg3133
added a commit
to garg3133/zulip
that referenced
this issue
Nov 7, 2021
…t tool. This commit fixes the rest of the issues reported in zulip#20108, first set of which were fixed in zulip#20195. Closes: zulip#20108
danielpyon
pushed a commit
to danielpyon/zulip
that referenced
this issue
Dec 1, 2021
This resolves the issues reported in zulip#20108, major chunk of which were due to the incomplete support for importing the livechat streams/messages in the tool. So, it's best not to import any livechat streams/messages for now until a complete support for importing the same is developed.
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello!
I installed Zulip from Git and tried to run the convert_rocketchat_data script.
Zulip Version: 5.0-dev-2500-g73710e1cf0
RocketChat Version: 4.0
Server Specifications:
OS: Ubuntu 20.04.3 LTS
CPU: 4
Memory: 16GB RAM
SWAP: 4GB
The migration looks like following:
The situation is that it seems that the converter fails at some point but I don't know where. I also tried to run it with traceback but no additional infos were shown.
The text was updated successfully, but these errors were encountered: