This script will merge the msgstore.db file with the old schema generated by
WPWhatsAppToAndroid script into an already existing msgstore.db file from a WhatsApp Android.
(For those who are moving from WP directly): If you still haven't registered your phone in Android's WhatsApp, this is the time to do so.
No changes to the code (like those needed on 'WPWhatsAppToAndroid') are needed (afaik) so it should work out of the box for all users, no matter their country.
This script was designed to work with the database schema used in 2.19.360 version (January 2020), so you should be already using in Android this version.
If you are using a much more recent version, I highly recommend you to take a look to the database sample under the samples folder and make the necessary changes to the source code to accomplish to them.
Nothing else apart of what you did already with the WPWhatsAppToAndroid script is required to run this one.
First, we need to extract the database of your Android installation. There are tools like Whapa and WhatsApp Xtract Key DB Extractor, but we need to 'reinsert' the database in our WhatsApp installation just after altering it with this script, and using those tools need the databases to be reencrypted and restored like a normal backup, a much cumbersome process.
We will be using a virtual machine (I will be using VMWare Workstation 15, although you can use Hyper-V, QEMU or VirtualBox, any hypervisor is valid as the process is very similar between all of them) with an Android system on it. If your cumputer it's not that powerful, you can use any other Android device that is rooted.
Head over to the Android-x86 project official's page. There you can find all the versions of Android ported for x86 processors. I personally used Android 9.0, so all the instructions will be based on this, but you can use any version you want as long as it's 32-bit ISO file. 64-bit versions doesn't work very well in virtual machines.
Here is the latest available version for download. Again, make sure to download Android-x86 32-bit ISO file
Create a virtual machine where the guest operating system is Other Linux 32-bit, attaching the ISO you previously downloaded to it. Set a big amount of storage for it (at least, the same amount of storage that your Android phone has. Set also a minimum of 2 GB of RAM and a bridged network adapter
Now, we can install Android x86 in the virtual machine. Power it on, navigate to Advanced Options > Auto-Installation and confirm that you want to continue. After finishing the installation, choose Reboot.
Running Android x86 in a VM needs some tweaks, so you need to force VESA mode first. Press e 2 times in the first option of the GRUB menu and add the following to the boot line (pay attention, there is an space before the line you need to write):
nomodeset xforcevesa
Press Enter and then b to boot. You must repeat this procedure in GRUB everytime you boot the VM, otherwise it won't boot
Don't forget to configure your Google Account. Install WhatsApp and Link2SD from Google Play. You should also enable developer options going to Settings > System > Tablet information and tapping multiple times on Compilation number.
Once developer options are enabled, you can enable there Always on screen while charging, as you won't be able to wake the VM if it goes to sleep.
The process is as follows. Follow them carefully if you don't want to lose any of the messages sent in the meantime:
- Make a normal WhatsApp Backup in Google Drive
- Turn on airplane mode
- Uninstall WhatsApp
- Install WhatsApp in the VM.
- Extract the database
- Make the backup
If you don't want to lose any of the messages, make the backup when you are sure that nobody will be talking during the 2 minutes it usually takes to be performed and uploaded.
- Turn on airplane mode
Once you finish the backup, turn on airplane mode and check if somebody sent you a message while you were doing the backup. If that's not the case, simply repeat step 1 until nobody doesn't send anything.
- Uninstall WhatsApp
If everything went perfectly until this step, uninstall WhatsApp from your phone, erasing all the application data (in case your phone prompts you if you want to keep the app's data). At this point, you can turn off airplane mode.
- Install WhatsApp in the VM
Make sure the machine is connected to the internet through the bridged network adapter. Install WhatsApp in the VM, restoring the backup you previously did.
Check that you're logged in the VM with the same Google account you used for the backup. Also, pay attention to the date of the creation of the backup, if it's from some days back, you did something else wrong.
After your backup is restored, wait for WhatsApp to restore all the media you have in your chats. Don't worry if you receive messages during this process (which can take a while), you won't lose any of them.
When 'Restore completed' appears in the top of your chat list, close the app and re-open it. If no other pop-up related to the backup restoring process appears, continue.
Open Link2SD and find WhatsApp. Click on it and then, in the 3-dot menu at the right, click on 'Freeze'. This disables WhatsApp completely (it won't even appear in the home screen), so any message you will receive during this time won't be lost and will arrive after we finish tweaking the database.
Now open 'Terminal Emulator'. Type su. Grant root access. Then, go to WhatsApp's database directory with the following command: cd /data/data/com.whatsapp/databases
There, run ls -l, and check that msgstore.db and many files starting with msgstore are located there. Copy the files to the local storage directory: cp msgstore* /storage/emulated/0.
You now need to get those files (normally 3) out of your guest Android VM into your host OS. I suggest you setting a SFTP server in your local network and use the bridged network adapter to connect to it, like I did (pretty straightforward to configure without any guide, but if you still need help, check here. You can even install the server in another VM. I used Cx File Explorer to move the files from/to the server, as it's the best file browser I know with SFTP support.
If you don't want the hassle and geeky way of doing it, you can simply upload the files to Google Drive or Telegram and download them in your computer.
Depending on the task the script is doing, different databases files will be asked. Here is a glossary of the databases that you will need, so you can have them at a hand. This documentation (and the script) will use the following terms from now on:
- Original msgstore.db: This the msgstore.db file you just obtained. It comes from an Android WhatsApp's installation
- msgstore.db (or msgstore.db from my other script): This is the msgstore.db file generated by
WPWhatsAppToAndroid - messages.db: The original Windows Phone's WhatsApp database.
- calls.db: The original Windows Phone's WhatsApp database that stores calls.
- temp.db: Database generated when running
WPWhatsAppToAndroid.
Close any other programs, and even shutdown the VM if your PC has less than 8 GB of RAM. For speeding up the process (and it even takes an hour in databases with 500k messages), I designed the script to perform all the manipulations in RAM first and, only when they are completed, in disk (which is much slower). The formula to know the peak RAM consumption of my script is the following:
Size of original msgstore.db * 3 = Peak RAM consumption
It's time to run merger.py (Mac/Linux/etc...) or merger.exe (Windows). The script goes through the following phases. You can skip any of the phases in order to go to the following one using CONTROL + C keyboard shortcut.
Although everything is explained inside the script, here you go a guide:
Requires original msgstore.db
This part commits all the pending changes that were not commited by WhatsApp and vacuums the database, in order to re-arrange the data it's inside it.
Requires original msgstore.db, msgstore.db and calls.db
This part moves call logs from WP to the Android database.
Requires original msgstore.db
This part changes mentions appearing in chats as @XXYYYYYYYYY (being X your country code and Y a phone number) to a clickable mention to your contacts. Above, how your mentions appear after using WinWazzapMigrator, below how they will after running this part of the script:
Requires original msgstore.db, msgstore.db and messages.db
This part adds all the messages that are present in you WP databases, but aren't in the Android database. If you ran WinWazzapMigrator, this is useful for adding the original media to your chat with a blurred appearance, like this example:
If you're using just my script to move from WP to Android without using WinWazzapMigrator, this is the part where your messages will be moved to the Android's schema.
Stickers, files and system messages (like 'X encryption key was changed', profile photo changed, etc...) will be excluded from this, as they don't have thumbnail info in the database and will crash WhatsApp (at least in my version, 2.19.360).
Requires temp.db, original msgstore.db
In this part, replies from your WP database to your Android database.
If you're moving directly and not using WinWazzapMigrator, you still need to run this.
Before/After:
Requires original msgstore.db
YOU MUST RUN THIS PART IF YOU DID 'ADD MISSING MEDIA AND MESSAGES', OR MESSAGES WILL APPEAR IN A COMPLETE DISORDER.
WhatsApp sorts their messages by their IDs in the database (instead of doing it by timestamp). This means that messages that were added to the 'messages' table before will appear before your most recent messages.
In this process, all the database is copied into RAM ordered by ascending timestamps, the data is deleted from the original database and they are inserted back, but in the correct order.
This also simplifies a litle bit further data analysis (if you're interested in that), as the messages will be in the 'messages' table completely and perfectly ordered.
This is the part of the script that is more prone to break by WhatsApp updates, as this part changes the references to the messages that are used by other tables. Fixing this should be as simple as inspecting which tables are in the original msgstore.db using a database viewer and add the appropiate UPDATE sentences in the code of my script if any of the tables that references messages aren't already in my code.
If you already ran all the other parts succesfully but you still didn't run this one, it's time for you to check this. If you already ran this and you didn't receive any errors, you can continue and re-import the database into WhatsApp, probably nothing wrong will happen. However, if WhatsApp added a table that references messages and it wasn't handled by my script, it might left some orphaned or corrupted data behind. You're advised to check this before continuing and repeating the execution of my script
This repeats the same process done in the
Briefly, in this part, we will do:
- Move the altered original msgstore.db back to the VM
- Turn ON airplane mode in the VM for checking thata everything in WhatsApp is in order
- Replace the installed database with the altered version
- Unfreeze WhatsApp and check results
- Perform a Google Drive backup
- Reinstall WhatsApp in your phone
Basically, all we need to do is repeat the same process we did before to extract the database, but in reverse.
Move your (already modified by the script) original msgstore.db back to the local storage directory of your VM, using the same procedure you did before.
As you might have noted, you copied three files, two of them didn't had the .db extension. Those files were partially commited data and tehy were already handled by the database, so forget about them. You only need to take care of the altered original msgstore.db
After moving the database back to the VM, you can isolate it completely from the exterior. This will help us to check our chats and see if WhatsApp doesn't crash and everything is working perfectly.
In the terminal, located in WhatsApp's database directory (/data/data/com.whatsapp/databases), run the following commands:
rm -rf msgstore*
cp /storage/emulated/0/msgstore.db .
Run ls -l. You will see an output similar to this:
Take a look at my example. You can see at the left the Unix permission schema and, after that the user and the group that owns the files. As you can see in my examples (ignoring all the files that doesn't end with .db), all the files have a permission set of rw-rw---- and the group and user that owns the file are u0_a81.
In my case, I will need to run the following commands to set the permissions correctly:
chown u0_a81:u0_a81 msgstore.db
chmod 660 msgstore.db
If you don't understand well how Unix permissions work and you see a really different permission set in the output, google first how to apply the correct permissions with chmod. The user/group set (u0_a81 in my case) is unique for every app you have installed, so you will see a different user/group set in your output for sure.
Change your command accordingly.
Verify again at this point that you have airplane mode enabled (or all connectivity disabled)
Open Link2SD, search for WhatsApp and choose 'Unfreeze' in the 3-dot menu in the top right.
Open WhatsApp now and check that everything works correctly but search. Searching messages won't work at this point, but they will in the following steps, so don't mind. Check that your messages are in order, that replies, call logs, thumbnails and so on were added, that you don't have any crashes in some chats, etc. If you open a group and you're able to scroll really fast to the top for 30 seconds without a crash, everything is correct.
Once you're sure that this is the state you want for your database, disable airplane mode and enable connectivity. Only when you're sure that this is the database which you would like to carry on from now on. This is a one-time confirmation that you can't revert back.
The messages you received while doing all of this will arrive correctly. I promised that you wouldn't lose any messages, and you didn't right?
Same as you did in the original Google Drive backup: make sure that in the timeframe you start the backup and you finish it you don't receive any messages, otherwise they will be lost if you don't run another backup.
This time the backup will be take much more time than the first one, and it will be stuck during some time at 100%. Don't worry, this is normal. WhatsApp is populating the search cache at this point, and this can take a lot if you have many messages.
Once the backup finishes and you're sure that nobody sent you a message during that timeframe, turn on again airplane mode in the VM. Then, uninstall WhatsApp in the VM. Reinstall it in your phone and configure it
Initial setup might be also longer than normal, as WhatsApp might be still working in populating the search cache.
Once you're logged in again in your phone, check that search works correctly.
If facing any issues, publish the exception code received. For seeing it, just reproduce what you were doing again in a cmd or bash terminal, executing the script from there (instead of double clicking on it in the file browser), as the script might simply crash.
IF YOU FACE AN ERROR, YOU ARE SUGGESTED TO NOT USE THE RESULTING MSGSTORE.DB DATABASE IN ANY WAY, AS YOU MIGHT END UP WITH AN INCOMPLETE OR INCOHERENT DATABASE
Make sure that you also follow all the steps carefully, there are many stuff that can be missed if you're not paying attention to my docs and what you're doing. Doing things incorrectly with this script could potentially and completely break your database.
Also, remember that disabling airplane mode in 2.5 step means a total and complete confirmation, with no coming back, that you agree with the state of the database. If you regret it later, you would lose all the messages you received after disabling airplane mode.