Android OTA payload.bin extraction #233

keesj-exset · 2022-04-20T13:02:45Z

Is your feature request related to a problem? Please describe.

"Modern" android OTA updates contain a file called payload.bin that start with a magic "CrAU"
While emba does find some part of the file it would be nice to have full support for this (common) file format

Describe the solution you'd like
Starting from a zip obtained from

https://developers.google.com/android/ota
oriole-ota-sd1a.210817.015.a4-19a77b62.zip
https://dl.google.com/dl/android/aosp/oriole-ota-sd1a.210817.015.a4-19a77b62.zip

git clone https://github.com/vm03/payload_dumper.git
cd payload_dumper
pip install -r requirements.txt

unzip ../oriole-ota-sd1a.210817.015.a4-19a77b62.zip
python payload_dumper.py payload.bin

Processing system partition..........................................................................................................................................................................................................................................................................................................................................................................................................................................Done
Processing system_ext partition..........................................................................................................................Done
Processing product partition.............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Done
Processing vbmeta_system partition.Done
Processing boot partition................................Done
Processing vendor_boot partition................................Done
Processing dtbo partition........Done
Processing vbmeta partition.Done
Processing vbmeta_vendor partition.Done
Processing vendor partition......................................................................................................................................................................................................................................Done
Processing vendor_dlkm partition...........................Done
Processing bl1 partition.Done
Processing pbl partition.Done
Processing bl2 partition.Done
Processing abl partition.Done
Processing bl31 partition.Done
Processing tzsw partition...Done
Processing gsa partition.Done
Processing ldfw partition..Done
Processing modem partition...................................Done

The result is found in output
file output/*.img

output/bl1.img:           data
output/bl2.img:           data
output/bl31.img:          data
output/boot.img:          Android bootimg, kernel (0x150d94), ramdisk (0x630)
output/dtbo.img:          data
output/gsa.img:           data
output/ldfw.img:          data
output/modem.img:         POSIX tar archive (GNU)
output/pbl.img:           Dyalog APL version 165.7
output/product.img:       Linux rev 1.0 ext2 filesystem data, UUID=8e66e769-b7a9-574e-b7d2-513c40dbb996, volume name "product" (extents) (large files) (huge files)
output/system_ext.img:    Linux rev 1.0 ext2 filesystem data, UUID=fc31cba1-4585-5da0-9700-ecedd28b80ec, volume name "system_ext" (extents) (large files) (huge files)
output/system.img:        Linux rev 1.0 ext2 filesystem data, UUID=02e80408-f118-591d-90f7-5b2411e5859b (extents) (large files) (huge files)
output/tzsw.img:          data
output/vbmeta.img:        data
output/vbmeta_system.img: data
output/vbmeta_vendor.img: data
output/vendor_boot.img:   data
output/vendor_dlkm.img:   Linux rev 1.0 ext2 filesystem data, UUID=d550f889-ddd7-5920-bf31-ffd8c5dee97e, volume name "vendor_dlkm" (extents) (large files) (huge files)
output/vendor.img:        Linux rev 1.0 ext2 filesystem data, UUID=d8891240-d867-5fac-80a5-6e9859e0263d, volume name "vendor" (extents) (large files) (huge files)

It would be nice to integrate this .

Describe alternatives you've considered

Manual work

Additional context
https://www.thecustomdroid.com/how-to-extract-android-payload-bin-file/

The text was updated successfully, but these errors were encountered:

m-1-k-3 · 2022-04-20T19:44:30Z

Looks quite interesting and should not too hard to integrate this feature into EMBA. Is EMBA currently able to extract something useful from such a payload file?

m-1-k-3 · 2022-04-20T20:48:13Z

If you have a full installation of EMBA up and running you could give it a try over here: https://github.com/m-1-k-3/emba

Please note that you currently have to install the needed deps manually and it is only running in dev mode (./installer.sh -F):

sudo python3 -m pip install protobuf
sudo pip install bsdiff4
cd external
git clone https://github.com/vm03/payload_dumper.git
cd ..
sudo ./emba.sh -f ./android/ota-payload.bin -l ./android/testlogs -D -S -t -s -z

In my initial testrun it looks not too bad. It extracts the ota update but fails with the ext extractor afterwards. I will take a deeper look into this issue the next days.

m-1-k-3 · 2022-04-30T12:10:23Z

Merged in master. Docker container updated.
Have phun and give us feedback :)

keesj-exset · 2022-05-16T07:24:31Z

Hi,

I tried the code on a ota.zip file and on a plain payload.bin. It looks like the OTA extractor only triggers on a top level payload.bin and not when inside a zip. I am not 100% sure because I also got the "Extractor needs too much disk space" message. anyway:

It looks like the installation of the dependencies is not yet working

==> Android OTA extractor
-----------------------------------------------------------------
00000000  43 72 41 55 00 00 00 00  00 00 00 02 00 00 00 00  |CrAU............|
00000010  00 01 f4 3f 00 00 02 0b  18 80 20 20 8b f4 b1 af  |...?......  ....|
00000020  07 28 8b 04 60 00 6a b3  bb 01 0a 06 73 79 73 74  |.(..`.j.....syst|
00000030  65 6d 10 01 1a 1b 73 79  73 74 65 6d 2f 62 69 6e  |em....system/bin|
00000040  2f 6f 74 61 70 72 65 6f  70 74 5f 73 63 72 69 70  |/otapreopt_scrip|
00000050  74 22 04 65 78 74 34 3a  28 08 80 80 d4 a9 03 12  |t".ext4:(.......|
00000060  20 41 a7 38 e6 90 a4 84  bc 64 bb 10 81 14 97 97  | A.8.....d......|
00000070  37 a9 1b 33 a8 bc 55 6c  2f b3 1e 59 f2 fd 97 ac  |7..3..Ul/..Y....|
00000080  6c 42 31 08 08 10 00 18  a4 c2 3a 32 05 08 00 10  |lB1.......:2....|
00000090  80 04 42 20 82 fe 45 9b  a4 1b 58 d9 92 8d c6 8e  |..B ..E...X.....|

[*] Extracting Android OTA payload.bin file ...

Traceback (most recent call last):
  File "/home/vagrant/emba/./external/payload_dumper/payload_dumper.py", line 7, in <module>
    import bsdiff4
ModuleNotFoundError: No module named 'bsdiff4'
find: '../log4/firmware/android_ota/': No such file or directory
find: '../log4/firmware/android_ota/': No such file or directory
[*] Extracted 0 files and 0 directories from the firmware image.
[*] Mon May 16 03:19:02 EDT 2022 - P25_android_ota finished
[*] Mon May 16 03:19:02 EDT 2022 - P60_firmware_bin_extractor starting

keesj-exset · 2022-05-16T07:25:02Z

[^[[0;33m*^[[0m] Fri May 13 07:06:38 EDT 2022 - P17_gpg_decompress finished
[^[[0;33m*^[[0m] Fri May 13 07:06:38 EDT 2022 - P18_qnap_decryptor starting
[^[[0;33m*^[[0m] Fri May 13 07:06:38 EDT 2022 - P18_qnap_decryptor finished
[^[[0;33m*^[[0m] Fri May 13 07:06:38 EDT 2022 - P19_bsd_ufs_mounter starting
[^[[0;33m*^[[0m] Fri May 13 07:06:38 EDT 2022 - P19_bsd_ufs_mounter finished
[^[[0;33m*^[[0m] Fri May 13 07:06:38 EDT 2022 - P25_android_ota starting
[^[[0;33m*^[[0m] Fri May 13 07:06:38 EDT 2022 - P25_android_ota finished
[^[[0;33m*^[[0m] Fri May 13 07:06:38 EDT 2022 - P60_firmware_bin_extractor starting
[^[[0;35m!^[[0m]^[[0;35m Fri May 13 07:10:09 EDT 2022 - Extractor needs too much disk space 13209^[[0m
[^[[0;35m!^[[0m]^[[0;35m Fri May 13 07:10:09 EDT 2022 - Ending extraction processes^[[0m
[^[[0;33m*^[[0m] Fri May 13 07:14:50 EDT 2022 - P60_firmware_bin_extractor finished
[^[[0;33m*^[[0m] Fri May 13 07:14:50 EDT 2022 - P65_package_extractor starting



┌──(vagrant㉿EMBAbox)-[~/emba]
└─$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            4.8G     0  4.8G   0% /dev
tmpfs           973M  888K  972M   1% /run
/dev/sda1       372G   62G  295G  18% /
tmpfs           4.8G     0  4.8G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           973M   64K  973M   1% /run/user/130
tmpfs           973M   60K  973M   1% /run/user/1000

└─$ du -hm "/home" --max-depth=1 --exclude="proc" 2>/dev/null | awk '{ print $1 }' | sort -hr | head -1 || true
28104

./emba.sh:  export MAX_EXT_SPACE=11000     # a useful value, could be adjusted if you deal with very big firmware images

230:
    if [[ "$DISK_SPACE" -gt "$MAX_EXT_SPACE" ]]; then
      print_output "[!] $(date) - Extractor needs too much disk space $DISK_SPACE" "main"
      print_output "[!] $(date) - Ending extraction processes" "main"
      DISK_SPACE_CRIT=1
      break
    fi

m-1-k-3 · 2022-05-16T14:32:56Z

Hi,

I tried the code on a ota.zip file and on a plain payload.bin. It looks like the OTA extractor only triggers on a top level payload.bin and not when inside a zip. I am not 100% sure because I also got the "Extractor needs too much disk space" message. anyway:

It looks like the installation of the dependencies is not yet working

==> Android OTA extractor
-----------------------------------------------------------------
00000000  43 72 41 55 00 00 00 00  00 00 00 02 00 00 00 00  |CrAU............|
00000010  00 01 f4 3f 00 00 02 0b  18 80 20 20 8b f4 b1 af  |...?......  ....|
00000020  07 28 8b 04 60 00 6a b3  bb 01 0a 06 73 79 73 74  |.(..`.j.....syst|
00000030  65 6d 10 01 1a 1b 73 79  73 74 65 6d 2f 62 69 6e  |em....system/bin|
00000040  2f 6f 74 61 70 72 65 6f  70 74 5f 73 63 72 69 70  |/otapreopt_scrip|
00000050  74 22 04 65 78 74 34 3a  28 08 80 80 d4 a9 03 12  |t".ext4:(.......|
00000060  20 41 a7 38 e6 90 a4 84  bc 64 bb 10 81 14 97 97  | A.8.....d......|
00000070  37 a9 1b 33 a8 bc 55 6c  2f b3 1e 59 f2 fd 97 ac  |7..3..Ul/..Y....|
00000080  6c 42 31 08 08 10 00 18  a4 c2 3a 32 05 08 00 10  |lB1.......:2....|
00000090  80 04 42 20 82 fe 45 9b  a4 1b 58 d9 92 8d c6 8e  |..B ..E...X.....|

[*] Extracting Android OTA payload.bin file ...

Traceback (most recent call last):
  File "/home/vagrant/emba/./external/payload_dumper/payload_dumper.py", line 7, in <module>
    import bsdiff4
ModuleNotFoundError: No module named 'bsdiff4'
find: '../log4/firmware/android_ota/': No such file or directory
find: '../log4/firmware/android_ota/': No such file or directory
[*] Extracted 0 files and 0 directories from the firmware image.
[*] Mon May 16 03:19:02 EDT 2022 - P25_android_ota finished
[*] Mon May 16 03:19:02 EDT 2022 - P60_firmware_bin_extractor starting

I will do a check on this during the next days ...

m-1-k-3 · 2022-05-17T06:37:54Z

As far as I can see bsdiff4 is missing on your system. In the docker container you will find a working installation of it. So, you can use the official docker image and everything should works as expected.

Regarding your local installation I am wondering as the bsdiff4 package should be installed via the installer module installer/IP00_extractors.sh

Could you please verify your local installation and the state of the package:

└─$ sudo pip list | grep bsdiff     
bsdiff4                            1.2.2

The deep extraction mode for ota updates is fixed in #246

m-1-k-3 added the enhancement New feature or request label Apr 20, 2022

m-1-k-3 added help wanted Extra attention is needed good first issue Good for newcomers labels Apr 20, 2022

m-1-k-3 self-assigned this Apr 20, 2022

m-1-k-3 removed help wanted Extra attention is needed good first issue Good for newcomers labels Apr 20, 2022

m-1-k-3 mentioned this issue Apr 28, 2022

Preparation PR for version 1.0 #234

Merged

m-1-k-3 closed this as completed Apr 30, 2022

m-1-k-3 reopened this May 16, 2022

m-1-k-3 added the bug Something isn't working label May 16, 2022

m-1-k-3 mentioned this issue May 17, 2022

Deep extraction of android ota update files #246

Merged

p4cx closed this as completed in #246 May 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Android OTA payload.bin extraction #233

Android OTA payload.bin extraction #233

keesj-exset commented Apr 20, 2022

m-1-k-3 commented Apr 20, 2022

m-1-k-3 commented Apr 20, 2022

m-1-k-3 commented Apr 30, 2022

keesj-exset commented May 16, 2022

keesj-exset commented May 16, 2022

m-1-k-3 commented May 16, 2022

m-1-k-3 commented May 17, 2022

Android OTA payload.bin extraction #233

Android OTA payload.bin extraction #233

Comments

keesj-exset commented Apr 20, 2022

m-1-k-3 commented Apr 20, 2022

m-1-k-3 commented Apr 20, 2022

m-1-k-3 commented Apr 30, 2022

keesj-exset commented May 16, 2022

keesj-exset commented May 16, 2022

m-1-k-3 commented May 16, 2022

m-1-k-3 commented May 17, 2022