Support for the new persistent map format

[Hypfer]

No description provided.

[leonardpitzu]

nvmap attached in new rr.gz format

navmapfirst.ppm.log.rr.gz

[JohnRev]

Some findings:

The static encryption key (RoCKR0B0@BEIJING) is no longer used.
Instead, a random key is generated for each encrypted file. The key is sourced from /dev/urandom.
Then, this key is itself encrypted using mbedtls_pk_encrypt. The first 512 bytes of the file represent the encrypted key.
The remaining file content past the first 512 bytes is the map encrypted with mbedtls_aes_crypt_ecb and the random key.

That's what I have found so far, but it would be cool if someone else can independently confirm this.

[JohnRev]

Here's another update.

The encrypted rr.gz files go through this process:

1. A random key is obtained by reading /dev/urandom/
2. The key is encrypted with an RSA public key, using mbedtls_pk_encrypt
3. The encrypted key is stored in the file header (first 512 bytes)
4. The gzip-ed file is encrypted using AES ECB mbedtls_aes_crypt_ecb and appended to the .rr.gz file

This is presumably so that Xiaomi can later decrypt the key using their RSA private key, which will then enable them to decrypt the gzip archive.

Some possible workarounds:

1. Intercept the calls to /dev/urandom and redirect to a custom file. This way, we will always know what the "random" key is.
2. Replace the public key with one which we know the private key for
3. Disable encryption all together.

Options 2, and 3 would break the compatibility with the Xiaomi cloud servers, and so they are best used with dummycloud enabled. Option 1 could theoretically still work, but can be easily detected by Xiaomi.

I tried experimenting with Option 1 for a bit, but was unsuccessful. So I decided to just disable encryption, since I do not care about the Xiaomi cloud because my vacuum is not allowed to connect to the internet anyway. Turns out this is the simplest option.

I updated my gen2 vacuum to 001780 and patched rrlogd. The patched file is attached here.
Then, I updated the regex that I pointed out in an earlier comment: #40 (comment)
Valetudo now correctly shows the maps on v001780 when the vacuum is docked.

The patched rrlogd for v001780 is attached here. It should be copied to the vacuum and moved to /opt/rockrobo/rrlog/. Don't forget to give it execution permission, chmod +x /opt/rockrobo/rrlog/rrlogd
Note: it is probably wise to keep a copy of the original rrlogd file first: mv /opt/rockrobo/rrlog/rrlogd /opt/rockrobo/rrlog/rrlogd_.
Also note: that I did not test this on any other version, and it will probably not work and require different patch offsets.
Final note: You will need to either reboot your vacuum, or kill the running rrlogd process first so that the robot will re-spawn a process with the patched rrlogd.

Here are the hex patches used:

0001051C: 4B 47
00010746: FF E3
00010747: F7 20

rrlogd_patched_001780_gen2.zip

[leonardpitzu]

@JohnRev can you patch a rrlog (of course i will privide it) for gen2? i would try this and see how the new persistant pams work out...by the way - what's the setting needed to enable the map storage feature?

[JohnRev]

This is gen2. I should have clarified.
Not sure about the map storage. I was mainly focused on finding means for decrypting the rr.gz file :)

[leonardpitzu]

gr8. i'll download 1780 and get myself to work. i guess we should e carefull and stop the automatic update mechanism - xiaomi will most probably close some doors and we might remain stuck with some default stuff...

[JohnRev]

My vacuum has never connected to the internet, so I'm not too worried about the automatic updates.
If you do use the patched rrlogd file, it would probably be wise to use dummycloud too as I'm pretty sure the patched rrlogd breaks compatibility with the xiaomi cloud services.

[leonardpitzu]

i was connected to the internet (i got the gadget exactly when it came on the market). now i am on dummycloud and ccrypt is removed (just an empty file). my firewall is also plugging all holes so i should be safe. have you tried to enable the map storage feature? and if yes how did you do it?

[JohnRev]

No, sorry, I haven't looked into the map storage feature. Tho if you do figure out how to enable it, let me know!

[leonardpitzu]

@JohnRev i updated the beast and used your patched rrlogd but i keep getting error 500 from the webserver...

[JohnRev]

@leonardpitzu, you will have to compile Valetudo with the updated regex from #40 (comment)

[leonardpitzu]

@JohnRev Already done that - I get the correct path to the files and I am trying correctly to open them. That’s where I got with the debugging. I user the rrlogd you provided. Added some more logging to figure out what’s wrong...

[JohnRev]

Interesting. It works fine for me. How does your /mnt/data/rockrobo/rrlog/ folder look like?
If there are a bunch of subfolders there, i suggest you delete them and try again with a fresh sweep of the vacuum.
Then, after the vacuum is docked, there should be a new subfolder containing a bunch of .gz file. Can you confirm they are unencrypted gzip files?

[JohnRev]

Oh, and did you reboot your vacuum to make sure it uses the patched rrlogd?
If you dont want to reboot, try killing the running rrlogd process (find process id with ps aux | grep rrlogd then kill -9 [pid]). The vacuum will respawn one with the patched rrlogd (assuming it is placed in the right path)

[dugite-code]

@JohnRev How did you go about patching rrlogd?

[leonardpitzu]

@JohnRev i've done some more debugging (i know linux quite well but i'm 250km away from the vacuum so debugging is a bit more tricky :) ). anyway here's what i found (again, using fw 1780 on gen 2, your patched rrlogd and a patched/modified version of valetudo):

• the webserver searches for the long folder names stored in /mnt/data/rockrobo/rrlog
• the webserver makes a list of folders and searches for maps (this time using the new regex from Error 500 when using map function while charging #40)
• if a folder and a mapfile and a logfile are found within the folder the map is being "reconstructed", that is the log is being decrypted.

and here it stops with the error:

digital envelope routines:EVP_DecryptFinal_ex:bad decrypt

Previousely the log was being encrypted with the standard symmetric key which we all know. So the webserver was decryptiong it using the key, unpacking it and do the rest of it's thing.
From your description i figured you disabled the encryption all together so the webserver now failes to decrypt the data. Even more, when i download the encrypted logs locally and try to unpack them it fails - probably the magic number is wrong.

[JohnRev]

@leonardpitzu :Just to confirm, did you make sure the running rrlogd process is the patched one? Simply replacing the file won't do it.
It sounds to me the gz files are still being generated by the unpatched rrlogd

[leonardpitzu]

@JohnRev copied the patched rrlogd and rebooted the thing... i'm doing some more testing to see what's going on...

[JohnRev]

Oh! I think I attached the wrong rrlogd. In the one you're using right now, the first 512 bytes are the encrypted key and the rest of the file is the (unencrypted) gzip. So if you do something like dd if="navmap702931.ppm.0001.rr.gz" of="navmap-unencrypted.gz" bs=1 skip=512 you should get the clean unencrypted gz.
I have updated my initial comment. If you don't mind, please use this rrlogd. It is the same one as I have on my vacuum right now.
md5sum is 93e8df884b4c6162ffa2a11161f47429
Send it to the vacuum, kill the running rrlogd process (or reboot), and try again.
rrlogd_patched_001780_gen2.zip

[JohnRev]

@dugite-code: I disassembled rrlogd, looked for the function call that appends the encrypted key to the header, and disabled (NOP) that call. Then I looked for the call that stores the encrypted gz bytes to the file, and made them store the unencrypted buffer instead. The hex patches posted in my first comment do just that, and you can apply them to your unpatched rrlogd (of version 1780) to get the same patched rrlogd that I posted here.

[leonardpitzu]

@JohnRev i looked at the attached rrlogd and it's not the patched one. I patched it and am testing it now. will report back

[JohnRev]

Thanks! I am waiting. Sorry for attaching the wrong rrlogd initially ... I got my files mixed up 😄

[leonardpitzu]

@JohnRev sudo "you deserve a beer!" it's working!

[JohnRev]

Awesome!! And thanks for being willing to testing it 😄

I discovered there's some 500 errors still showing up after some time, but these are strictly related to the code logic of how the /mnt/data/rockrobo/rrlog is scanned in FIND_LATEST_MAP_IN_ARCHIVE function. I will write about them in #40 .

Glad it's working, and hope this helps others too!

[JohnRev]

@leonardpitzu, I have a question for you.
After rebooting the vacuum, does the robot forget the last map when performing a new sweep and generates a new one? Or does it trace correctly over the previously generated map?

[leonardpitzu]

@JohnRev mine also has Alzheimer and generates a new map. This is a "feature" i would guess unless we figure out what the "save map" button actually does. If we can set the same flag, by hand, we can keep the map between reboots. I read somewhere that the setting is some new file in /mnt/data/rockrobo. The file has to contain the value 1 in order to enable the map storage feature. Yet i can't seem to find that post nowhere so i am stuck right now...

[herrwusel]

I just updated to 1780 and still get 500 errors with valetudo 0.9. Is it just me?

[cryptomilk]

Did you patch rrlogd?

[herrwusel]

I did when building the firmware but will try again today. Maybe I did something wrong there.

[cryptomilk]

./imagebuilder.sh --patch-rrlogd then check the log that it has been patched. My latest changes to the image builder script allow to already add valetudo to the flashable firmware image.

[herrwusel]

Here is what I did:

sudo ./dustcloud/devices/xiaomi.vacuum/firmwarebuilder/imagebuilder.sh -k id_ed25519.pub -f v11_001780.pkg --disable-xiaomi --patch-rrlogd
Scriptpath: ./dustcloud/devices/xiaomi.vacuum/firmwarebuilder
.../dc/imagebuilder.sh
compatible readlink found!
Generate SSH Host Keys if necessary
decrypt soundfile
ccrypt: .../dc/english.pkg: Kennwort passt nicht -- unverändert
unpack soundfile
decrypt firmware
ccrypt: .../dc/v11_001780.pkg: Kennwort passt nicht -- unverändert
unpack firmware
patch ssh host keys
disable SSH firewall rule
integrate SSH authorized_keys
reconfiguring network traffic to xiaomi
pack new firmware
encrypt firmware
2ef9c67f1a7d9b6a9ea292837e4bbf2a  output/v11_001780.pkg

I don't read anything from the rrlogd patch.

[cryptomilk]

I don't see that it patches rrlogd so is the git branch up to date and do you have bspatch installed?

[herrwusel]

I was missing the bsdiff package. I got this now:

./dustcloud/devices/xiaomi.vacuum/firmwarebuilder/imagebuilder.sh -k id_ed25519.pub -f v11_001780.pkg --disable-xiaomi --patch-rrlogd
Scriptpath: ./dustcloud/devices/xiaomi.vacuum/firmwarebuilder
.../dc/imagebuilder.sh
compatible readlink found!
Generate SSH Host Keys if necessary
decrypt soundfile
ccrypt: .../dc/english.pkg: Kennwort passt nicht -- unverändert
unpack soundfile
decrypt firmware
ccrypt: .../dc/v11_001780.pkg: Kennwort passt nicht -- unverändert
unpack firmware
patch ssh host keys
disable SSH firewall rule
integrate SSH authorized_keys
reconfiguring network traffic to xiaomi
checking if we can patch rrlogd
pack new firmware
encrypt firmware
bce3c1cdf5fdd16f14b0209636a8538e  output/v11_001780.pkg

After pushing the firmware to to vacuum I still get the error 500. How do you add valetudo while building?

[Hypfer]

Did you start a new cleanup? Since you were running an unpatched rrlogd it might be that the data it's trying to display is still encrypted.

The removed encryption only applies to new cleanups.

[JohnRev]

In addition to what @Hypfer pointed out, it seems that your rrlogd patch didn't work?
In the log you pasted, I can only see checking if we can patch rrlogd, but no follow-up .. it should be followed by creating backup of rrlogd and patching rrlogd.
On your vacuum, can you do md5sum /opt/rockrobo/rrlog/rrlogd, if it is not 93e8df884b4c6162ffa2a11161f47429, then it is not the patched rrlogd for gen2 v1780.
You can replace it manually with this one https://github.com/Hypfer/Valetudo/files/2670885/rrlogd_patched_001780_gen2.zip and kill the running rrlogd process (or reboot the vacuum).
See earlier posts for more details.

[herrwusel]

Pushing the patched rrlogd manually did the trick - thanks @JohnRev and @Hypfer. Still dont know why the image wasnt build correctly.

[JohnRev]

Other things stored in PersistData_1.data include: "Goto Routing", "Clean Routing" ( paths/routes?), "Dock position", "Robot position". Haven't had time to update my code for parsing them ... might do that if I get some free time :)

[Fietspomp86]

I don't know if this belongs here, but I also get a 500 error when charging. When roborock (gen1, version 3514 rooted) is running, map is generated just fine.
When charging the roborock/api/map/latest shows: Error: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt

So does mine also need a patched rrlogd?

[ralf-ms]

I'm having the same problem as Fietspomp86, during cleaning the generated map can be viewed without any difficulties, as soon as the robo has finished cleaning, the map vanishes and the above html error 500 or with "../latest" error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt) is displayed.

[cryptomilk]

It is possible that on the gen1 firmware version, rrlogd has a new encryption method. However someone needs to reverse engineer it to see what actually is going on.

[JohnRev]

@Fietspomp86 rrlogd for gen1 will need to be patched too, but none have been published yet. @ralf-ms are you also using gen1 with 3514 firmware?

[ralf-ms]

yes, correct: gen1 / 3514

[Fietspomp86]

Ok allright, what do I need to do then? I'm fairly ok with linux so I can try to patch it with some instructions.

[JohnRev]

Try the attached rrlogd, if you want. I haven't tested it myself as I don't have a gen1 vacuum.

gen1_v11_003514_rrlogd_patch.zip

[Fietspomp86]

Ok, I've copied the rrlogd and rebooted. Apparently that removed Valetudo which is strange. Installed Valetudo again and started vacuum to test. Now it's turning off and on every 10 seconds.
And right now it's speaking Chinese to me.... So for me it's not working yet....

EDIT: Strange enough it has resetted itself to old firmware I guess, speaking Chinese and disconnected from WiFi.

[JohnRev]

I am guessing the "patched" rrlogd failed and ended up crashing the robot, which then went into recovery mode... maybe?
Sorry about that. As I said, I don't have the gen1 to test, and I just tried applying the same patches that I made for gen2 onto the gen1 rrlogd (while adjusting the offsets accordingly).
I may look at this more if I get some time

[JohnRev]

After transfering the rrlogd to the robot, did you make sure you gave it execution permissions?
chmod +x /opt/rockrobo/rrlog/rrlogd

[ralf-ms]

The patched rrlogd is working. I've just tried a cleaning session and could retrieve the map after the finished session.
Thank you very much, JohnRev!

[JohnRev]

No problem!
I will try to post an "auto patcher" script that would patch rrlogd on any firmware/generation.

[Hypfer]

Can we please not use this issue or any issue at all for basic tech support.

This is not a forum but rather a place to gather information and use said information to implement the feature mentioned in the title.

[Fietspomp86]

After transfering the rrlogd to the robot, did you make sure you gave it execution permissions?
chmod +x /opt/rockrobo/rrlog/rrlogd

Quite possible I forgot that.... I'll retry it when it's back online.

[Hypfer]

Oh ffs.

[JohnRev]

@Hypfer: This thread should probably be closed. As you pointed out, the conversation has gotten out of hand.
The new map format parsing is now well understood. The rrlogd patches (in gen1 or gen2) have already been documented. No changes are required in Valetudo itself. Maybe just update the README.md ?

There is another issue created for the newer features that come with the persistent maps #72