[WhatsApp Web Beta] Stuck on getIdbObjects

[thewh1teagle]

I tried to use your script with chromium to store the session on file but after I logged in using the qr code the scripts hangs on.
As you can see in the picture, I use the beta version of whatsapp.

[TSRBerry]

Oh that's a wierd issue, I don't know how this is happening yet, but I'll figure it out soon.
Can you tell me which option you tried to use? I don't think that it actually matters for the bug, but maybe it'll help.

That's the second issue related to WhatsApp Web Beta now, so they may have made some changes that trigger some funky behavior within my script. Can anyone participate in this or is there some special requirement for WhatsApp Web Beta?

[thewh1teagle]

I tried to use it with chrome and the option to save the session to local file.
Also, I have a plan to make this thing as browser extension.
I tried to manually backup and restore indexeddb user object store in wac database, and the localstorage. But it was not enough, whatsapp does not signed in. Maybe we can compare my indexeddb databases versus yours (because i use the beta version) then i will be able to figure out what databases to backup

[TSRBerry]

That sounds great!

But I would prefer not to compare these databases publicly since they contain keys for authentication to WhatsApp.
Since you are developer too and have some experience with IndexedDB and LocalStorage, I want to tell you about the stuff to leave out and then you could provide the data that WhatsApp Web Beta stores in there.
I will follow up with another message as soon as I looked up the exact names and tell you how to look for the things I think I need.

[TSRBerry]

Okay, let's do that a bit differently.

Log in to WhatsApp Web Beta and look through LocalStorage and IndexedDB. Look for a key that says something like WASecretBundle. Normally there should also be WAToken1 and WAToken2 around there, so if you find them in the same place that's good news.

I don't need the exact values of these keys. I just need to know where they are.
Normal WhatsApp Web stores these keys for IndexedDB this way: DB "wawc" -> ObjectStore "user" so you would find all the important keys and values in there.

Could you tell me where these keys are stored in the beta version for IndexedDB?
If you also find them in localStorage, please tell me if they still use these keys directly.

[thewh1teagle]

I can't see any key like WASecretBundle or WAToken1 or WAToken2 in indexedDB

Some useful info

My local storage keys

> Object.keys(localStorage)
[
  'whatsapp-mutex',
  'remember-me',
  'MdUpgradeWamFlag',
  'WARoutingInfo',
  'theme',
  'syncd_disabled_due_to_fatal',
  'system-theme-mode',
  'WANoiseInfoIv',
  'XXXXXXXXXXXXXXXX==',
  'debugCursor',
  'XXXXXXXXXXXXXXXXXXXXX==',
  'contact-sync-refresh-seconds',
  'last-wid-md',
  'WaInitialHistorySynced',
  'MdHistoryLastChunkProcessed',
  'WALangPref',
  'XXXXXXXXXXXXXXXXXXX==',
  'XXXXXXXXXXXXXXXXXXX==',
  'preserved_user_keys',
  'md-opted-in',
  'mobile-platform',
  'XXXXXXXXXXXXXXXXXXXXX==',
  'XXXXXXXXXXXXXXXXXXXXX==',
  'WANoiseInfo',
  'WAHistorySyncStatus',
  'XXXXXXXXXXXXXXXXXXXXX==',
  'XXXXXXXXXXXXXXXXXXXXX==',
  'XXXXXXXXXXXXXXXXXXXXX==',
  'XXXXXXXXXXXXXXXXX==',
  'WAShouldCheckContactSyncStatus',
  'abprops',
  'XXXXXXXXXXXXXX==',
  'WebEncKeySalt',
  'old-logout-cred',
  'XXXXXXXXXXXXXXXXXXXXXX==',
  'XXXXXXXXXXXXXXXXXXXXXX==',
  'XXXXXXXXXXXXXXXXXXXXXX==',
  'history-sync-earliest-date',
  'XXXXXXXXXXXXXXXXXXXXXX==',
  'critical_data_synced'
]

IndexedDB databases

user objectStore inside wawc database in indexedDB

[
    {
        "key": "XXXXXXXXXXXXXXXXXXXXXX==",
        "value": "false"
    },
    {
        "key": XXXXXXXXXXXXXXXXXXX==",
        "value": "{\"XXXXXXXXXXXXXXXXXX==\":1,\"XXXXXXXXXXXXXXXXXXXX==\":2}"
    },
    {
        "key": "XXXXXXXXXXXXXXXXXXX==",
        "value": "{\"DELETE_MSG_CLEAR_MEDIA\":true,\"MD_PAYMENT\":true,\"ARCHIVE_BROADCAST\":true,\"RECENT_EMOJI_SYNC\":true,\"VOIP_VOICE_CALL\":true,\"DESKTOP_VOIP_VOICE_CALL\":true,\"DESKTOP_VOIP_VIDEO_CALL\":true,\"MD_BACKEND\":true,\"MD_ADV\":true,\"MD_VOIP_GROUP\":false,\"VOIP_INDIVIDUAL_OUTGOING\":true,\"GROUPS_V_3\":true,\"GROUPS_V_3_CREATE\":false,\"CHANGE_NUMBER_V_2\":true,\"QUERY_STATUS_V_3_THUMBNAIL\":false,\"LIVE_LOCATIONS\":true,\"QUERY_VNAME\":true,\"VOIP_INDIVIDUAL_INCOMING\":true,\"PAYMENTS\":true,\"STICKER_PACK_QUERY\":true,\"LIVE_LOCATIONS_FINAL\":true,\"MEDIA_UPLOAD\":true,\"VNAME_V_2\":true,\"VIDEO_PLAYBACK_URL\":true,\"STATUS_RANKING\":true,\"VOIP_INDIVIDUAL_VIDEO\":false,\"THIRD_PARTY_STICKERS\":true,\"FREQUENTLY_FORWARDED_SETTING\":true,\"GROUPS_V_4_JOIN_PERMISSION\":true,\"RECENT_STICKERS\":false,\"CATALOG\":true,\"STARRED_STICKERS\":false,\"TEMPLATE_MESSAGE\":true,\"TEMPLATE_MESSAGE_INTERACTIVITY\":false,\"EPHEMERAL_MESSAGES\":true,\"RECENT_STICKERS_V_2\":true,\"USER_NOTICE\":true,\"SUPPORT\":true,\"GROUP_UII_CLEANUP\":true}"
    },
    {
        "key": "XXXXXXXXXXXXXXXXXXXXXXXXX==",
        "value": "[]"
    },
    {
        "key": "XXXXXXXXXXXXXXXXXX==",
        "value": "false"
    },
    {
        "key": "WALangPref",
        "value": "\"en\""
    },
    {
        "key": "WALogPreemptiveCleanUp",
        "value": "false"
    },
    {
        "key": "WARoutingInfo",
        "value": "{\"domain\":\"fb\",\"edgeRouting\":\"XXXXXX==\"}"
    },
    {
        "key": "WAShouldCheckContactSyncStatus",
        "value": "true"
    },
    {
        "key": "XXXXXXXXXXXXXXXXXXXX==",
        "value": "[0,17]"
    },
    {
        "key": "debugCursor",
        "value": "1222"
    },
    {
        "key": "XXXXXXXXXXXXX==",
        "value": "{\"XXXXXXXXXXXXXX==\":1}"
    },
    {
        "key": "mobile-platform",
        "value": "\"android\""
    },
    {
        "key": "XXXXXXXXXXXXXXXXXXXX==",
        "value": "{\"id\":\"global_mute\",\"expiration\":0}"
    },
    {
        "key": "whatsapp-mutex",
        "value": "\"xXXXXXXXXX:init_XXXXXXXXXXXXX\""
    },
    {
        "key": "XXXXXXXXXXXXXXXXX==",
        "value": "false"
    }
]

[TSRBerry]

Thank you for providing so much information!
That's really interesting. At first glance it seems like they are using a new format now and the keys I should store are in localStorage again.

I will update my script to allow WhatsApp Web Beta to work with it and tell you when you can test the new version. This shouldn't take too long, so I think I'll be done in about 2 hours since I can't work on it right away.

[thewh1teagle]

I already tried to manually restore the whole localStorage and user objectStore inside wawc database but whatsapp didn't connect.
Do you have another way that should work?

[TSRBerry]

Could you describe your process to me?
Maybe I am doing something different, but all my script should do at the moment, is to dump localStorage/IndexedDB to a file after verifying that it found a valid WhatsApp Web login (which is where the script is stuck for you). To load a session it first opens WhatsApp Web normally, imports the file and reloads the page. After that you should be logged in again.
Is this similar to what you did?

[thewh1teagle]

Yes. that's what I tried to do.
You can see what I tried here

[TSRBerry]

Hmm, since I don't have access to the beta, it'll be a lot harder to test.

I would say, let's try that with my implementation again and if that doesn't work I'll ask you to send me the other databases like you did with the wawc database earlier.
It'll be a lot of try and error, but maybe we can get it to work again. If you have other ideas feel free to tell me, so we can talk about them as well.

[TSRBerry]

I pushed my changes (8d13394), which means you could try to run my script again and see if it works (we would need a lot of luck though).
Please show me the output of the script if something fails and hopefully we can figure something out.

[thewh1teagle]

I tried your latest update by unfortunately it doesn't worked

screenshot

Can we continue talk in telegram?
https://t.me/joinchat/5ewKXry5-TQzMjMx

[thewh1teagle]

Finally I got it to work
Now I need to find the specific databases for backup instead all

poc here

/** Promisified IndexedDB Request */
const preq = req =>
    new Promise((resolve, reject) => {
        req.addEventListener("error", reject);
        req.addEventListener("success", e => resolve(e.target.result));
    });

/** Promisified IndexedDB upgrade Request */
// const preupgrade = req =>
//     new Promise((resolve, reject) => {
//         req.addEventListener("error", reject)
//         req.addEventListener("upgradeneeded", e => resolve(e.target.result))
//     })

async function dumpIndexedDB() {
    const result = {};
    const dbs = await window.indexedDB.databases();
    for (const { name, version } of dbs) {
        const info = result[name] = {
            ver: version,
            tables: {},
        };
        const db = await preq(indexedDB.open(name));
        const tables = Array.from(db.objectStoreNames);
        console.log(tables)
        const trans = db.transaction(tables, "readonly");
        for (const tableName of tables) {
            const table = trans.objectStore(tableName);
            // add indexes
            var indexes = {}
            for (const indexName of table.indexNames) {
                var index = table.index(indexName)
                indexes[index.name] = {keyPath: index.keyPath, multiEntry: index.multiEntry, unique: index.unique}
            }
            info.tables[tableName] = {
                keyPath: table.keyPath,
                autoIncrement: table.autoIncrement,
                indexes: indexes,
                data: await preq(table.getAll()),
            };
        }
    }
    return result
}

async function restoreIndexedDB(dump) {
    for (let dbName in dump) {
        // delete old db
        console.log(`deleting db ${dbName}`)
        indexedDB.deleteDatabase(dbName)
        // var db = await preupgrade(indexedDB.open(dbName, 1))
        await new Promise((resolve, reject) => {
            
            console.log(`opening new db ${dbName} version ${dump[dbName]['ver']}`)
            var request = indexedDB.open(dbName, dump[dbName]['ver'])
            request.onerror = e => reject(request)
            request.onupgradeneeded = e => {
                var db = e.target.result
                for (const tableName in dump[dbName]['tables']) {
                    // debugger
                    var tableInfo = dump[dbName]['tables'][tableName]
                    // create table
                    console.log(`creating table ${tableName} in db ${dbName}`)
                    if (!tableInfo.keyPath) tableInfo.keyPath = "key"
                    var table = db.createObjectStore(tableName, { keyPath: tableInfo['keyPath'], autoIncrement: tableInfo['autoIncrement'] } )
    
                    // restore table indexes
                    for (const indexName in tableInfo['indexes']) {
                        console.log(`restoring index ${indexName} in table ${tableName} in db ${dbName}`)
                        const indexInfo = tableInfo['indexes'][indexName]
                        table.createIndex(indexName, indexInfo.keyPath, indexInfo)
                    }
    
                    // restore table data
                    console.log(`restoring data for table ${tableName}`)
                    for (const row of tableInfo['data']) {
                        table.add(row)
                    }
                }
                db.close()
                resolve("ok")
            } 
        })
    }
    return "ok"
}

async function deleteData() {
    // localStorage.clear()
    for (let db of await indexedDB.databases()) {
        console.log(`deleting db ${db.name}`)
        indexedDB.deleteDatabase(db.name)
    }
}

async function test() {
    var data = await dumpIndexedDB()
    await restoreIndexedDB(data)
    console.log("done")
}

[TSRBerry]

I tried your latest update by unfortunately it doesn't worked
I would have expected it to stop after a while instead of trying unlimted times. That's odd. I will look over my code again and see if I can do something about that.

Can we continue talk in telegram?
https://t.me/joinchat/5ewKXry5-TQzMjMx
I don't have telegram, so I can't really contact you there, but I prefer to stay on here anyways.

Finally I got it to work
Now I need to find the specific databases for backup instead all
Nice! Thank you for also providing the poc. I'll take a look and see if I can think of a way to detect what I need to provide to WhatsApp Web to get it to load the session again. If you now the specific DBs or ObjectStores that are required feel free to post them here! That would speed up the process of adding support for the beta by a lot!

Thank you again for your work!

[TSRBerry]

Looking at the db names from your reply before (#15 (comment)) I would narrow down the list of necessary dbs like that:

• Try to restore: wawc, wawc_db_enc, signal-storage
  -> If that works try it without signal-storage
• Otherwise add fts-storage and model-storage
  -> If that works try it without signal-storage
  --> If that worked as well try to leave out fts-storage or model-storage next.
  --> In case it didn't work you can only try to leave out fts-storage or model-storage or both and maybe one of these cases still works.

Then we have a list of DBs that have to be included and can start poking at ObjectStores. I can't do a lot without having the ObjectStores and their keys, so it would be great if you could post the keys for the other DBs and OSs here as well. That way I could at least give you some kind of script that would figure out what we would need to save and help you with that!

[thewh1teagle]

Tried, It worked when I restore
wawc + wawc_db_enc + signal_storage
And of course the localStorage as well.

now another thing -
the session is restored but the chats is empty, and whatsapp not trying to re-sync messages.

[TSRBerry]

Nice! Have you tried doing it without localStorage? In the normal version WhatsApp Web would only check if one of them is there and because they used to switch between those two places I just restore the information to both localStorage and IDB.

Ahh I think I might have read something about that before... Take a look at page 10 of this preview security whitepaper. I didn't read all of it yet, but they might hint at what they are using to resync message history and stuff. (I found that whitepaper on this page if you are curious: https://www.whatsapp.com/security/ )

As a first shot, there is this WaInitialHistorySynced key in localStorage that you could set to false when restoring the session. Maybe that will trigger a resync immediately. If this doesn't work look for the same or a similar key in IDB. We should be able to trigger it on our own somehow, so that key would be my first guess. Also if you go through the logs that are stored in IDB do they tell you something about syncing chats and contacts?

[thewh1teagle]

When I keep only indexeddb without localStorage it's not working, whatsapp throws me out.
Tried to set
WaInitialHistorySynced, WAHistorySyncStatus, in indexeddb, and in localstorage (they are in both), but that's not triggering resync at all.
Maybe we should call the function directly.
I found the function but it's bundeled with webpack, and the function has parameters and I don't know how to call it correctly.

[thewh1teagle]

Update:
I found that when I set critical_data_synced key to false inside wawc db with user objectStore,
whatsapp trying to resync the messages (I guess)
But it tries forever. after some time when I set it back to true and reload the page, whatsapp working but the data isn't restored.

This is the "restoring proccess" that never finished

[TSRBerry]

I finally got the option to join the Multi Device Beta!

So I will try to get it working with my script and while I am at it, I want to implement some general changes to the script to make it more reliable.

Hopefully I will be able to solve this issue soon!

[thiagofsr97]

Awesome news @jeliebig , I can't wait for your updates with Multi Device to be available. Hope you can get it working as soon as possible.

[LinkedTech]

@jeliebig Have you found a solution for the Multi Device? Look forward to your reply.

[TSRBerry]

Currently not at home, but was planning on finishing it today!

Hopefully I can do it and don't have to let you guys wait any longer.

[LinkedTech]

@jeliebig Thanks for your great effort!

[TSRBerry]

Unfortunately I was not able to finish it this weekend.
I am almost done though, just need to fix a few bugs my IndexedDB methods before it completely works.

Since Firefox doesn't support indexedDB.databases() I needed to think about another method that I could use to get the db names. Although I am not really happy with my solution for this problem, I am pretty sure it will work.

Sorry to keep you guys waiting, I will comment here again, once everything is fixed and tested.

But for now I need to get some sleep.

[thewh1teagle]

@jeliebig
Create an array of the db names then indexedDB.databases will not be necessery

[TSRBerry]

That's what I kind of did. I am still rewriting a lot of this project and want to implement everything specific to WhatsApp in it's own subclass.
I was hoping to come up with a trick, so I could use a general approach to dump IDB easily. But since that is not option without indexedDB.databases() I instead created a list of DBs to dump from the contents of __dbnames.
After everything works, I will try to minimize the amount of data that needs to be saved. But for now it is more important to get it to work at all.

Currently the rewrite works for dumping the session, but I still have some bugs in the restore process.
This is the last thing that needs to work before I can merge it into master.

@thewh1teagle I used your PoC from above as a reference for my new __set_indexed_db() method. Could I include you in a new Credits section in the README.md file? I already included some comments that point to your PoC in code comments (see: my last commit), but I would like to credit you more visibly, since not everybody will look at the code.

[thewh1teagle]

Thanks for asking, Yes you can and I would like to : - )
About the restore point - as I remember my poc working with restore, I just backuped the whole indexdb + localStorage + cookies
but indexedDB is pretty heavy because it has a lot of whatsapp media

[amirbokaei]

Hi, I also had this problem
If there is a problem, I can help

[TSRBerry]

Hi, I also had this problem
If there is a problem, I can help

This issue will be solved soon.
I tried to finish it last weekend, but ran into some bugs with IDB.

There is one annoying issue remaining that I need to fix, before I can merge everything into master.

I am sorry that it takes me such a long time, but I currently need to study a lot for my exams and don't have as much time for my projects as I usually have.

[TSRBerry]

There is one annoying issue remaining that I need to fix, before I can merge everything into master.

Nevermind, this will take longer than I expected.

After fixing the remaining issue, I tested the script again and saw that my session wasn't restored correctly and WhatsApp would not let me login. This led to some more testing and I figured out which IDB databases are required to restore a working session from WhatsApp Web. One of these is wawc_db_enc and it contains two CryptoKeys which are marked as not extractable.

To get this working, I need to figure out when the key gets imported and how the site receives it.
Or another really bad idea would be to compile a Chromium/Firefox version that doesn't care about the extractable property, so the key could be exported with that and since importing shouldn't be a problem, I guess it would work. But that doesn't sound like a good way to deal with this problem, so I will need more time to figure this out.

[amirbokaei]

thank you for your effort

In which branch is the modified project now? (I know it is not finished yet)

[TSRBerry]

Everything is currently in this branch: partial-rewrite

[LinkedTech]

@jeliebig Hi, Whatsapp is going to phase out the current web version and upgrade to the multi-device version in 7 days. See if there is a working version / workaround that we can use. Thanks a lot for your great effort.

[TSRBerry]

If that's true, I don't think there is anything that I could do to get the script to work again.

The problem I'm facing is that I can't export the necessary CryptoKey from the browser, since it's marked as non-exportable.
It might be possible to get the key by reverse engineering WhatsApp js files, but I currently don't have any interest in that.
Aside from that, I also think it would be out of scope for this tool, as I rely on selenium to handle browser things and only provide a way to export the session as a small file.

This project was mostly a learning experience for me and I wanted to learn how open source works (even though I only got a tiny glimpse here, it was very valuable to me).
I never expected that people would really be interested in this simple application, but I'm glad it was useful to someone. It's a little sad that it ends this way, but nothing can be done about it. I hope people will be interested in my future projects, because there are a lot of things I want to do and just need to find the time to actually do them. :D

I'm going to leave this issue open in case I or anyone else wants to work on it, but for now I don't think there is anything I can do.
I will confirm in a separate comment when WhatsApp switches to the new multi-device version for everyone and update the readme at that point.

Thanks for helping me with this project and thank you so much for using it!

[TSRBerry]

WhatsApp started rolling out their new multi-device version this week and thus my script will stop working as soon as people update.

I updated the README accordingly and will stop working on this project.

Thank you guys for helping me (especially with this issue) and for using my script until it stopped working!