Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tablet won't talk to fake server after downgrade #7

Open
jebediah2 opened this issue Jun 7, 2021 · 17 comments
Open

Tablet won't talk to fake server after downgrade #7

jebediah2 opened this issue Jun 7, 2021 · 17 comments

Comments

@jebediah2
Copy link

Running the hack from a Mac's Terminal on a RM2 with the latest firmware 2.7.1.53, but all it does is it keeps saying I'm up to date. I think I've done all the steps as required but there seems to be no desired effect when running update_engine_client -check_for_update or pressing the "Check for updates" box in the UI.

  • I placed the 2.5.1.47_reMarkable2.signed update file in the /updates folder
  • I modified the update.conf file with SERVER=http://my-host-name:8000 and also tried with my IP SERVER=http://my.ip.address:8000
  • I ran python3 serve.py with and without changing line 46 to host = "my.ip.address"
    Output:
    Device should use:  http://<built-in function gethostname>:8000/
    Available updates: {'reMarkable2': ('2.5.1.47', '2.5.1.47_reMarkable2.signed')}
    Starting fake updater: 8000
  • Depending on the tinkering, I would also get Device should use: http://my.ip.address:8000/ or Device should use: http://my-host-name:8000/ on the first line
  • On the RM2 I ping my host with port 8000 to make sure it's reachable
  • I enable automatic updates in the UI then run update_engine_client -check_for_update
    Output:
    I20210607 12:31:08.343346 384 update_engine_client.cc:271] Initiating update check and install.
  • If I am on the Software screen on the RM2 at the same time, I can see that running the command triggers an update check ("Checking..."), which then displays a check mark with message "Your device is up to date."

So all looks like the RM2 is talking to the legit remarkable's server and reports that the installed version is the latest.

Looking at the logs:

root@reMarkable:~# journalctl -u update-engine -f
-- Logs begin at Mon 2021-06-07 11:57:58 UTC. --
Jun 07 12:34:26 reMarkable update_engine[188]:     <ping status="ok"/>
Jun 07 12:34:26 reMarkable update_engine[188]:   </app>
Jun 07 12:34:26 reMarkable update_engine[188]: </response>
Jun 07 12:34:26 reMarkable update_engine[188]: I20210607 12:34:26.105146   188 omaha_request_action.cc:444] No update.
Jun 07 12:34:26 reMarkable update_engine[188]: I20210607 12:34:26.105720   188 action_processor.cc:99] ActionProcessor::ActionComplete: finished OmahaRequestAction, starting OmahaResponseHandlerAction
Jun 07 12:34:26 reMarkable update_engine[188]: I20210607 12:34:26.106050   188 omaha_response_handler_action.cc:38] There are no updates. Aborting.
Jun 07 12:34:26 reMarkable update_engine[188]: I20210607 12:34:26.106323   188 action_processor.cc:81] ActionProcessor::ActionComplete: OmahaResponseHandlerAction action failed. Aborting processing.
Jun 07 12:34:26 reMarkable update_engine[188]: I20210607 12:34:26.106623   188 action_processor.cc:87] ActionProcessor::ActionComplete: finished last action of type OmahaResponseHandlerAction
Jun 07 12:34:26 reMarkable update_engine[188]: I20210607 12:34:26.106885   188 update_attempter.cc:313] Processing Done.
Jun 07 12:34:26 reMarkable update_engine[188]: I20210607 12:34:26.107182   188 update_attempter.cc:351] No update.

Out of curiosity I tried running switch.sh on the RM2 and I get this:

root@reMarkable:~# ./switch.sh
new: 2
fallback: 3

Is there something I'm missing or is it just not going to work with this specific downgrade?

@ddvk
Copy link
Owner

ddvk commented Jun 7, 2021

after switch.sh you have to reboot

also when you set the server ip, did you remove the # from the line?

@jebediah2
Copy link
Author

jebediah2 commented Jun 7, 2021

also when you set the server ip, did you remove the # from the line?

Well that was embarrassing because no I just assumed the default update.conf parameters were being used so I didn't even spot the #

Have corrected that now and RM2 is finally talking to the host server. However I am getting an error Unable to get http response code: error:1408F10B:SSL routines:ssl3_get_record:wrong version number on device side, while on server side it says code 400, message Bad request version

Looks like I'm having an HTTPS/SSL issue?

Here are the full logs:

RM2

Jun 07 14:53:00 reMarkable update_engine[188]: I20210607 14:53:00.701758   188 dbus_service.cc:62] Attempting interactive update
Jun 07 14:53:00 reMarkable update_engine[188]: I20210607 14:53:00.701864   188 update_attempter.cc:283] New update check requested
Jun 07 14:53:00 reMarkable update_engine[188]: I20210607 14:53:00.714282   188 omaha_request_params.cc:75] Current group set to Prod
Jun 07 14:53:00 reMarkable update_engine[188]: I20210607 14:53:00.714942   188 update_attempter.cc:533] Not updating boot flags, we don't know the state of the application.
Jun 07 14:53:00 reMarkable update_engine[188]: I20210607 14:53:00.715014   188 update_attempter.cc:710] Scheduling an action processor start.
Jun 07 14:53:00 reMarkable update_engine[188]: I20210607 14:53:00.715250   188 action_processor.cc:41] ActionProcessor::StartProcessing: OmahaRequestAction
Jun 07 14:53:00 reMarkable update_engine[188]: I20210607 14:53:00.716625   188 omaha_request_action.cc:280] Posting an Omaha request to https://192.168.1.20:8000
Jun 07 14:53:00 reMarkable update_engine[188]: I20210607 14:53:00.716711   188 omaha_request_action.cc:281] Request: <?xml version="1.0" encoding="UTF-8"?>
Jun 07 14:53:00 reMarkable update_engine[188]: <request protocol="3.0" version="2.7.1.53" requestid="{d9be6e49-719f-447f-80fa-61f01b5a4c84}" sessionid="{7d93afaa-a01d-47d7-a6bd-91569cb1d7b1}" updaterversion="0.4.2" installsource="ondemandupdate" ismachine="1">
Jun 07 14:53:00 reMarkable update_engine[188]:     <os version="codex 3.1.3" platform="reMarkable2" sp="2.7.1.53_armv7l" arch="armv7l"></os>
Jun 07 14:53:00 reMarkable update_engine[188]:     <app appid="{98DA7DF2-4E3E-4744-9DE6-EC931886ABAB}" version="2.7.1.53" track="Prod" ap="Prod" bootid="{6f969ef1-3452-431a-a0a3-072fca3b341e}" oem="RM110-043-15440" oemversion="3.1.3" alephversion="2.7.1.53" machineid="76088fffa3574c29a3fa4eed681a1e29" lang="en-US" board="" hardware_class="" delta_okay="false" nextversion="0.0.0" brand="" client="" >
Jun 07 14:53:00 reMarkable update_engine[188]:         <ping active="1"></ping>
Jun 07 14:53:00 reMarkable update_engine[188]:         <updatecheck></updatecheck>
Jun 07 14:53:00 reMarkable update_engine[188]:         <event eventtype="3" eventresult="2" previousversion=""></event>
Jun 07 14:53:00 reMarkable update_engine[188]:     </app>
Jun 07 14:53:00 reMarkable update_engine[188]: </request>
Jun 07 14:53:00 reMarkable update_engine[188]: I20210607 14:53:00.716758   188 libcurl_http_fetcher.cc:50] Starting/Resuming transfer
Jun 07 14:53:00 reMarkable update_engine[188]: I20210607 14:53:00.716959   188 libcurl_http_fetcher.cc:176] Setting up curl options for HTTPS
Jun 07 14:53:00 reMarkable update_engine[188]: I20210607 14:53:00.717721   188 libcurl_http_fetcher.cc:466] Setting up timeout source: 1 seconds.
Jun 07 14:53:00 reMarkable update_engine[188]: E20210607 14:53:00.856253   188 libcurl_http_fetcher.cc:264] Unable to get http response code: error:1408F10B:SSL routines:ssl3_get_record:wrong version number
Jun 07 14:53:00 reMarkable update_engine[188]: I20210607 14:53:00.856626   188 libcurl_http_fetcher.cc:281] No HTTP response, retry 1
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.710574   188 libcurl_http_fetcher.cc:50] Starting/Resuming transfer
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.714782   188 libcurl_http_fetcher.cc:176] Setting up curl options for HTTPS
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.716008   188 libcurl_http_fetcher.cc:466] Setting up timeout source: 1 seconds.
Jun 07 14:53:10 reMarkable update_engine[188]: E20210607 14:53:10.883882   188 libcurl_http_fetcher.cc:264] Unable to get http response code: error:1408F10B:SSL routines:ssl3_get_record:wrong version number
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.884632   188 libcurl_http_fetcher.cc:295] Transfer resulted in an error (0), 0 bytes downloaded
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.884968   188 libcurl_http_fetcher.cc:297] Error buffer: error:1408F10B:SSL routines:ssl3_get_record:wrong version number
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.885212   188 omaha_request_action.cc:670] Omaha request response:
Jun 07 14:53:10 reMarkable update_engine[188]: E20210607 14:53:10.885429   188 omaha_request_action.cc:686] Omaha request network transfer failed.
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.885660   188 action_processor.cc:81] ActionProcessor::ActionComplete: OmahaRequestAction action failed. Aborting processing.
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.885872   188 action_processor.cc:87] ActionProcessor::ActionComplete: finished last action of type OmahaRequestAction
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.886077   188 update_attempter.cc:313] Processing Done.
Jun 07 14:53:10 reMarkable update_engine[188]: E20210607 14:53:10.886341   188 update_attempter.cc:685] Update failed.
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.886560   188 utils.cc:750] Converting error code 2000 to kActionCodeOmahaErrorInHTTPResponse
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.886773   188 payload_state.cc:104] Updating payload state for error code: 37 (kActionCodeOmahaErrorInHTTPResponse)
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.886984   188 payload_state.cc:110] Ignoring failures until we get a valid Omaha response.
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.887537   188 action_processor.cc:41] ActionProcessor::StartProcessing: OmahaRequestAction
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.890130   188 omaha_request_action.cc:280] Posting an Omaha request to https://192.168.1.20:8000
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.890699   188 omaha_request_action.cc:281] Request: <?xml version="1.0" encoding="UTF-8"?>
Jun 07 14:53:10 reMarkable update_engine[188]: <request protocol="3.0" version="2.7.1.53" requestid="{6a389cca-51f5-44bd-94f5-796ceae2d7a6}" sessionid="{7d93afaa-a01d-47d7-a6bd-91569cb1d7b1}" updaterversion="0.4.2" installsource="ondemandupdate" ismachine="1">
Jun 07 14:53:10 reMarkable update_engine[188]:     <os version="codex 3.1.3" platform="reMarkable2" sp="2.7.1.53_armv7l" arch="armv7l"></os>
Jun 07 14:53:10 reMarkable update_engine[188]:     <app appid="{98DA7DF2-4E3E-4744-9DE6-EC931886ABAB}" version="2.7.1.53" track="Prod" ap="Prod" bootid="{6f969ef1-3452-431a-a0a3-072fca3b341e}" oem="RM110-043-15440" oemversion="3.1.3" alephversion="2.7.1.53" machineid="76088fffa3574c29a3fa4eed681a1e29" lang="en-US" board="" hardware_class="" delta_okay="false" nextversion="0.0.0" brand="" client="" >
Jun 07 14:53:10 reMarkable update_engine[188]:         <event eventtype="3" eventresult="0" errorcode="268437456"></event>
Jun 07 14:53:10 reMarkable update_engine[188]:     </app>
Jun 07 14:53:10 reMarkable update_engine[188]: </request>
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.892042   188 libcurl_http_fetcher.cc:50] Starting/Resuming transfer
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.892622   188 libcurl_http_fetcher.cc:176] Setting up curl options for HTTPS
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.893646   188 libcurl_http_fetcher.cc:466] Setting up timeout source: 1 seconds.
Jun 07 14:53:10 reMarkable update_engine[188]: E20210607 14:53:10.968775   188 libcurl_http_fetcher.cc:264] Unable to get http response code: error:1408F10B:SSL routines:ssl3_get_record:wrong version number
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.972926   188 libcurl_http_fetcher.cc:295] Transfer resulted in an error (0), 0 bytes downloaded
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.974318   188 libcurl_http_fetcher.cc:297] Error buffer: error:1408F10B:SSL routines:ssl3_get_record:wrong version number
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.974360   188 omaha_request_action.cc:670] Omaha request response:
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.974409   188 action_processor.cc:78] ActionProcessor::ActionComplete: finished last action of type OmahaRequestAction
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.974442   188 action_processor.cc:87] ActionProcessor::ActionComplete: finished last action of type OmahaRequestAction
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.974470   188 update_attempter.cc:313] Processing Done.
Jun 07 14:53:10 reMarkable update_engine[188]: I20210607 14:53:10.974512   188 update_attempter.cc:317] Error event sent.

Host server

Device should use:  http://<built-in function gethostname>:8000/
Available updates: {'reMarkable2': ('2.5.1.47', '2.5.1.47_reMarkable2.signed')}
Starting fake updater: 8000
192.168.1.33 - - [07/Jun/2021 16:53:00] code 400, message Bad request version ('\x9e·\x96')
�c0=ÅU?ã]ÆîÙ²'´é �·�" 400 -21 16:53:00] "üwý�4�	il|.;°0X
192.168.1.33 - - [07/Jun/2021 16:53:10] code 400, message Bad request version ("\x99¤XÆ\x14\x07ïÔßH\x8asÒH¹¶P8@-\x04²Xx\x9bXKÙ¢!]¢\x00\x96\x13\x02\x13\x03\x13\x01À,À0\x00£\x00\x9f̨̩̪À¯À\xadÀ£À\x9fÀ]ÀaÀWÀSÀ+À/\x00¢\x00\x9eÀ®À¬À¢À\x9eÀ\\À`ÀVÀRÀ$À(\x00k\x00jÀsÀw\x00Ä\x00ÃÀ#À'\x00g\x00@ÀrÀv\x00¾\x00½À")
192.168.1.33 - - [07/Jun/2021 16:53:10] "üÈ@F[bÜ:Cp	�Î�mÃ�¨²u#{å�T�g �¤XÆïÔßH�sÒH¹¶P8@-²Xx�XKÙ¢!]¢�À,À0£�̨̩̪À¯À­À£À�À]ÀaÀWÀSÀ+À/¢�À®À¬À¢À�À\À`ÀVÀRÀ$À(kjÀsÀwÄÃÀ#À'g@ÀrÀv¾½À" 400 -
192.168.1.33 - - [07/Jun/2021 16:53:11] code 400, message Bad request version ('v\x15c\x0f\x7f\x1a["\x00\x96\x13\x02\x13\x03\x13\x01À,À0\x00£\x00\x9f̨̩̪À¯À\xadÀ£À\x9fÀ]ÀaÀWÀSÀ+À/\x00¢\x00\x9eÀ®À¬À¢À\x9eÀ\\À`ÀVÀRÀ$À(\x00k\x00jÀsÀw\x00Ä\x00ÃÀ#À\'\x00g\x00@ÀrÀv\x00¾\x00½À')
192.168.1.33 - - [07/Jun/2021 16:53:11] "ü÷Õî
                                             Uþ·ÌÌ?þ¤ªPõmt5Á"¶R*Ú ¹Ç»á×7QO<þ>¿NÉC=Qø'$vc["�À,À0£�̨̩̪À¯À­À£À�À]ÀaÀWÀSÀ+À/¢�À®À¬À¢À�À\À`ÀVÀRÀ$À(kjÀsÀwÄÃÀ#À'g@ÀrÀv¾½À" 400 -

@ddvk
Copy link
Owner

ddvk commented Jun 7, 2021

you should set the url to: http://

@jebediah2
Copy link
Author

jebediah2 commented Jun 7, 2021

Thanks, that did the trick. I understand it shouldn't be necessary, but it would probably a good idea to make it clearer that one needs to remove the#and change https to http as it's easy to overlook these details when going through the instructions.

I think I'm going to give up on this though because even though all the logs confirmed the update to 2.5.1.47 took place as expected and the device was waiting for reboot, running ./switch.sh and rebooting didn't result in the device running a different firmware. Maybe running the switch script was for coming back to the legit firmware, not to validate the downgrade like I thought. That part of the instructions wasn't clear to me.

Going back to the Software update menu, the 2.8 version was offered, so I proceeded with that before making a new attempt with the downgrade. The modified update.conf and hosts files had been recovered to their stock version so I had to modify them again (taking care with the # and http), but strangely the RM2 is still quering https://remarkable-staging.auto-up.date/service/update2 instead of my host server. I double checked the host is reachable and that update.conf on the RM2 is modified as required. If that doesn't make the RM2 look for my server then I wonder what would.

@ddvk
Copy link
Owner

ddvk commented Jun 7, 2021

the switch.sh is just for changing parition, and is to be used separately (just to go to the other version)
after an update with the server you just have to reboot

you are on the beta channel and i suppose there is a different config or line for the update server

@jebediah2
Copy link
Author

after an update with the server you just have to reboot

So do you see any reason why the device didn't reboot into the downgraded firmware even though both server and device reported it had been successfully performed?

you are on the beta channel and i suppose there is a different config or line for the update server

I never enrolled for the beta releases. In fact I was never offered this release until after I rebooted following the 'successful' downgrade. Is it possible that the process allowed the device to query remarkable servers for beta releases even though it hadn't been configured to do so? Even now that I'm on 2.8.0.86, I can still press the button "Read more and enroll" and accept the user agreement, which I assume would no longer offer me to enroll had I already done that in the past.

Besides, the update.conf file still follows the exact same structure as the last stable 2.7, like so:

[General]
#REMARKABLE_RELEASE_APPID={98DA7DF2-4E3E-4744-9DE6-EC931886ABAB}
SERVER=http://192.168.1.20:8000
#GROUP=Prod
#PLATFORM=reMarkable2
REMARKABLE_RELEASE_VERSION=2.8.0.86

So yeah I don't really see why it still queries the official repo when looking for an update.

@jebediah2
Copy link
Author

jebediah2 commented Jun 8, 2021

The address being queried is remarkable-staging.auto-up.date (instead of get-updates.cloud.remarkable.engineering). I can block that domain from the hosts file but redirecting it to my host server IP won't work. Any idea where I should look in the file system to edit the server address the same way it was done in update.conf ?

I still can't figure how did the remarkable suddenly decide to query the beta server right after I had 'successfully' downgraded and rebooted, thus allowing me to perform the beta update and now locking me out of using the fake server.

@ddvk
Copy link
Owner

ddvk commented Jun 8, 2021

try uncommenting the REMARKABLE_RELEASE_APPID

@jebediah2
Copy link
Author

Thank you, though that didn't seem to change anything.

I noticed that when turning the update engine on, whether from the terminal or the UI, it seems to already know about the latest beta release and the URL to get it from, as well as the payload size and hash. Does it mean I have to trick it into forgetting about this information it seems to have memorised?

Jun 08 20:00:49 reMarkable systemd[1]: Starting Update Engine...
Jun 08 20:00:49 reMarkable update_engine[511]: I20210608 20:00:49.489493   511 main.cc:97] reMarkable Update Engine starting
Jun 08 20:00:49 reMarkable update_engine[511]: I20210608 20:00:49.491472   511 payload_state.cc:377] Current Response Signature =
Jun 08 20:00:49 reMarkable update_engine[511]: NumURLs = 1
Jun 08 20:00:49 reMarkable update_engine[511]: Url0 = https://eu-central-1.linodeobjects.com:443/remarkable-staging-2/build/reMarkable%20Device/reMarkable2/2.8.0.86/2.8.0.86-reMarkable2.signed
Jun 08 20:00:49 reMarkable update_engine[511]: Payload Size = 68377342
Jun 08 20:00:49 reMarkable update_engine[511]: Payload Sha256 Hash = xzSvlYK1kusX1RtR2aAMqY5qvRqISHRiXa1ZlKToFVc=
Jun 08 20:00:49 reMarkable update_engine[511]: Is Delta Payload = 0
Jun 08 20:00:49 reMarkable update_engine[511]: Max Failure Count Per Url = 10
Jun 08 20:00:49 reMarkable update_engine[511]: Disable Payload Backoff = 1
Jun 08 20:00:49 reMarkable update_engine[511]: I20210608 20:00:49.493404   511 payload_state.cc:402] Payload Attempt Number = 1
Jun 08 20:00:49 reMarkable update_engine[511]: I20210608 20:00:49.495187   511 payload_state.cc:429] Current URL Index = 0
Jun 08 20:00:49 reMarkable update_engine[511]: I20210608 20:00:49.496783   511 payload_state.cc:454] Current URL (Url0)'s Failure Count = 0
Jun 08 20:00:49 reMarkable update_engine[511]: I20210608 20:00:49.498322   511 payload_state.cc:488] Backoff Expiry Time = 01/01/70 00:00:00 UTC
Jun 08 20:00:49 reMarkable systemd[1]: Started Update Engine.
Jun 08 20:00:49 reMarkable update_engine[511]: I20210608 20:00:49.510005   511 update_check_scheduler.cc:82] Next update check in 9m47s

@danschrage
Copy link

I have this same issue, more or less (though I have an RM1). I was on 2.7.0.51 and wanted to downgrade to 2.5.1.47. In my case, the downgrade was successful. But I forgot to immediately shut off automatic updates, and without realizing it, it auto-updated to 2.8.0.86 right after I booted into 2.5.1.47. It prompted for a reboot, and when I did, it was back and running 2.8.0.86.

I successfully used switch.sh to go back to 2.5.1.47, and that works fine. But the fallback partition still contains 2.8.0.86. I tried updating (using serve.py) a couple times--to 2.5.0.27 and to 2.6.2.75. In both cases, though, it queries https://remarkable-staging.auto-up.date/service/update2 (as @jebediah2 reports), and then it re-downloads and installs 2.8.0.86. (Unlike @jebediah2, for me it always re-downloads 2.8.0.86; it doesn't report that it's already up to date, even though it just installs it on top of the existing 2.8.0.86 on the fallback partition.)

I don't want to clutter this issue if mine turns out to be subtly different, but I hope it's helpful to have a second case where update_engine_client seems to sidestep the serve.py server and query a different remarkable server directly. And a second case where I seem to have gotten a beta version without asking for it (as with @jebediah2, it still gives me the option to enroll in the beta in Settings).

I also have the same result as @jebediah2 where the update engine already seems to know about the latest release--that does seem to be a possible source of this issue.

@jebediah2
Copy link
Author

I noticed all the information displayed when turning the update engine on is contained in var/lib/update-engine/prefs/current-response-signature so I replaced everything with details from the 2.5.1.47 update (URL, file size and hash). Sure enough, next time I started the update engine, all that new information showed up, but that didn't affect the rest of the process. In the end, the tablet remains stubborn and queries the staging server, which consistently responds with an offer for 2.8.0.86. It looks like we'll be stuck until we find a way to force the tablet to query the stable server.

I also naively tried redirecting remarkable-staging.auto-up.date to my host's fake server but the latter can't respond since the tablet makes the request using https.

@jebediah2
Copy link
Author

jebediah2 commented Jun 10, 2021

Posting my experiments to get out of this deadlock.

When editing update.conf with a REMARKABLE_RELEASE_VERSION lower than 2.2.0.57, the staging server will offer an update to 2.2.0.57. It the stated version is 2.2.0.57 or higher, it will offer the 2.8.0.86 version and realise there's nothing to do.

The 2.2.0.57 update process is later denied after failing to verify the payload using the public key at /home/root/.update_engine/update-payload-key.pub.pem

I did this after enrolling then de-enrolling from the beta, hoping it would no longer query the beta server, but that has yet to change.

EDIT: Turns out, if you delete both public keys at
/home/root/.update_engine/update-payload-key.pub.pem
/usr/share/update_engine/update-payload-key.pub.pem
the verification is skipped and the update proceeds! I guess that's our downgrade, though to an old beta.

@danschrage
Copy link

I tried deleting the public keys (there was a third copy in / for me) and setting to version 1.0.0.0. But it still grabbed the 2.8.0.86 version and went ahead with the update.

I looked at /usr/sbin/update_engine (which is what gets started by update-engine.service). It did have one hardcoded URL (https://get-updates.cloud.remarkable.engineering/service/update2) that I thought could be the more general server it queries and gets redirected to https://remarkable-staging.auto-up.date/service/update2, so I tried patching the string in the executable, replacing it with my local serve.py server address (since I could also substitute http for https), but that had no effect. (I thought about modifying serve.py to use HTTPS with a self-signed certificate; maybe I'll try that later.) For me it downloads the 2.8.0.86 firmware before it ever queries https://remarkable-staging.auto-up.date/service/update2, so there must be some other place that it stores info about available updates.

I think that was similar behavior to what you were seeing: as soon as "Starting Update Engine..." appears in the logs, it immediately prints out the URL with the 2.8.0.86 update. Incidentally, that did not happen after I deleted the public keys--but only that once. After another update, the public keys are still gone, but it again prints out the URL to the 2.8.0.86 version as soon as the update engine starts, before it ever queries the remarkable-staging.auto-up.date server.

I also wondered whether 2.5.1.47 is a beta release? That was just a 2.5.x version I found by changing the URLs on the Remarkable update server. It seems like 2.5.1.45 was more widely deployed (e.g., it has a ddvk-hacks patch available while 2.5.1.47 does not). I wonder if update_engine is different for the beta releases and that's why the server is offering beta versions without enrolling in the beta. I can't find evidence of anyone else using this version, and that's one weird commonality for both of us.

@danschrage
Copy link

I solved my problem the old-fashioned way: by wiping everything and starting fresh. I used ddvk's uuuflash tool (https://github.com/ddvk/remarkable-uuuflash) to wipe the device and put 2.1.1.3 back on both partitions. It's quick--just get the tablet into recovery mode and then ./uuu reflash.uuu. But my RM is only three days old, so it was easy for me to start over (having learned everything I wished I had known three days ago to now set things up properly) without worrying about any data or settings.

After reflashing, I used serve.py to update to 2.6.2.75. That worked, and I then used serve.py to "downgrade" to 2.5.0.27, which also worked. I wanted to verify that it would accept a lower version as an update, and it does. And if I need to, I can use switch.py to switch between 2.5 and 2.6.

My best guess continues to be that there's something weird about 2.5.1.47, so perhaps it should be avoided. (I found some people complaining about problems with the updater in 2.5.1.45, so perhaps .47 changed something about the updater.)

@jebediah2
Copy link
Author

I'm really glad you could get your case sorted. I never realised there was a way to flash the remarkable, but that's because I'm looking at things with a RM2 filter, where all the tools haven't been developed yet. Putting the RM2 into recovery mode requires hardware (and Linux) and I'm not seeing the uuuflash tool as confirmed for the RM2 so I'll have to stay away for now.

I thought all this weird behaviour was coming from using the fake server more than from 2.5.1.47 itself, but you might be right. I now have 2 beta partitions, one on 2.8.0.86 and one on 2.2.0.57. I would expect 2.2 to query the right server but it still hangs on to the beta server, but it's hard to believe 2.5.1.47 left some persistent change to update-engine. But what do I know.

More weirdness, neither of my partitions offer updates anymore. I was expecting 2.2 to offer 2.8, but it doesn't and merely tells me I'm up to date. Thinking I'll just live with 2.2 for now, I re-installed Toltec on that partition and it instantly bricked the tablet. I later managed to re-apply the 2.2 update and not going to try to install much until I find a fix to move on to a newer, stable release.

Firmware URLs always use the same format and someone compiled a good deal of them in a reddit post. When I try to get 2.5.1.45 (for RM1 or RM2) it doesn't seem available. Maybe 2.5.1.47 was beta and that's why it wants to use the beta server just like those 2.2 and 2.8. That's how I thought of enrolling and de-enrolling from the beta (though no button for that on 2.2) Maybe comparing the update-engine binaries and other related files between 2.5.0.27 and 2.5.1.47 could tell us.

My only thought currently is to use the host file to redirect remarkable-staging.auto-up.date to the fake server. This usually leads to a no response error, but it might be different if I manage to perform what you suggested earlier by editing server.py and using a self-signed certificate.

@jebediah2 jebediah2 changed the title Am I missing something? RM2 talks to legit server instead of host's Tablet won't talk to fake server after downgrade Jun 11, 2021
@jebediah2
Copy link
Author

jebediah2 commented Jun 12, 2021

With help from @danschrage, I finally found the problem, and the fix. Both very simple.

It was probably correct to speculate that the 2.5.1.47 update was the problem. Whether it's really the case, downgrading to that version will make changes to /home/root/.config/remarkable/xochitl.conf and introduce (or activate) the dreaded beta server URL:
SERVER=https://remarkable-staging.auto-up.date/service/update2.
This setting seems to override the same line found in the usr/share/remarkable/update.conf file mentioned in the readme.

So all you have to do is edit xochitl.conf by changing the SERVER setting to your host (I used my local IP), not forgetting to change https to http. In update.conf, I left the SERVER line commented. Also, though it probably wasn't important, I had changed the REMARKABLE_RELEASE_VERSION to something lower (2.2.0.57) than the update I wanted to apply (2.5.0.27).

Once you've prepared all this, the fake server can be run normally and perform the downgrade as expected. I was coming from 2.8.0.86.

Once done and rebooted, xochitl.conf was unchanged this time, so it's probably a good idea to revert the SERVER setting to remarkable's genuine, stable release server:
SERVER=https://get-updates.cloud.remarkable.engineering/service/update2

@ddvk I suggest you mention this fix in the readme in case people get stuck with querying the beta server after updating to a bad version. Also, as mentionned in the beginning, it would probably a good idea to make it clearer that one needs to remove the # and change https to http on update.conf's SERVER setting, as it's easy to overlook these details when going through the instructions.

@ddvk
Copy link
Owner

ddvk commented Jun 12, 2021

ah, good catch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants