Utilizing more space for eMMC installation with ampart / 使用ampart来在eMMC安装中使用更多空间 #689

7Ji · 2022-11-10T08:10:08Z

This tool is a gem for amlogic's TV boxes with only a few gigabytes of emmc space. Not only does it use more space, it also reduces concerns about writing to unsafe data areas.

https://github.com/7Ji/ampart/releases/download/v1.0/ampart-aarch64-static.xz

Download and unzip, upload to Armbian system in USB. After booting from USB, execute the following command to adjust the partition.

01. Resize partition
./ampart-aarch64-static /dev/mmcblk2 --mode dclone data::-1:4
02. If the adjustment is successful, it will output information:
CLI write EPT: Write successful
03. Modify the [ armbian-install ] of the a script, set your own device id to use the full partition
   elif [[ "${boxid}" -eq "xxx" ]]; then
       BLANK1="117"
       BOOT="256"
       BLANK2="0"

Originally posted by @ophub in #620 (comment)

ampart is a partition tool I wrote earlier this year for HybridELEC and EmuELEC to replace the closed-source ceemmc. It supports reading and editting Amlogic's proprietary eMMC partition table (shortened as EPT, which co-exists with MBR/GPT partition table on eMMC) and the partitions node in stock DTBs, and has since become the one and only partition tool for EPT

You can either download its static AArch64 release or build from source on Armbian (which is more suggested, as the releases could be out-of-date). It works both on raw eMMC device and corresponding dumps so you can try on dumps obtained with dd for testing. (It also works on DTBs, so you can modify the partitions in an Android burning image directly, but I won't cover that use case here)

With a command like this, as posted by ophub:

./ampart-aarch64-static /dev/mmcblk2 --mode dclone data::-1:4

You can tell ampart to adjust the DTB and EPT and move the essential partitions you could not overwrite to the beginning of the eMMC, to like this:

===================================================================================
ID| name            |          offset|(   human)|            size|(   human)| masks
-----------------------------------------------------------------------------------
 0: bootloader                      0 (   0.00B)           400000 (   4.00M)      0
    (GAP)                                                 2000000 (  32.00M)
 1: reserved                  2400000 (  36.00M)          4000000 (  64.00M)      0
    (GAP)                                                  800000 (   8.00M)
 2: cache                     6c00000 ( 108.00M)                0 (   0.00B)      0
    (GAP)                                                  800000 (   8.00M)
 3: env                       7400000 ( 116.00M)           800000 (   8.00M)      0
    (GAP)                                                  800000 (   8.00M)
 4: data                      8400000 ( 132.00M)       1d16800000 ( 116.35G)      4
===================================================================================

So now you can safely write on areas 4M-36M (the gap between bootloader and reserved), 100M-116M (the gap between reserved and env), and 117M to end (env starts at 116M, but usually only contain 64K of data, so starting at 1M after 116M is more than safe)

So the armbian-install script can be modified to use the eMMC space starting from 117M (if you do it manually, then the above free space could also be used, i.e. 4M-36M, 100M-116M):

   elif [[ "${boxid}" -eq "xxx" ]]; then
       BLANK1="117"
       BOOT="256"
       BLANK2="0"

Although ampart has its documentation in my repo which you should refer to for more advanced uses (e.g. directly editting EPT instead of editting DTB and update EPT from DTB, move/clone/delete partitions, etc). But I'll explain why argument data::-1:4 is used for this general case.

The main usage for dclone mode is to restore a snapshot taken in dsnapshot mode, which, on a HK1Box that has not touched by ampart before, is like this:

logo::8M:1 recovery::24M:1 misc::8M:1 dtbo::8M:1 cri_data::8M:2 param::16M:2 boot::16M:1 rsv::16M:1 metadata::16M:1 vbmeta::2M:1 tee::32M:1 vendor::320M:1 odm::128M:1 system::1856M:1 product::128M:1 cache::1120M:2 data::-1:4

With a command like this, ampart can restore the partitions in DTB to exactly like when the snapshot is taken:

ampart /dev/mmcblk2 --mode dclone logo::8M:1 recovery::24M:1 misc::8M:1 dtbo::8M:1 cri_data::8M:2 param::16M:2 boot::16M:1 rsv::16M:1 metadata::16M:1 vbmeta::2M:1 tee::32M:1 vendor::320M:1 odm::128M:1 system::1856M:1 product::128M:1 cache::1120M:2 data::-1:4

Each partition argument is in the [name]:[offset]:[size]:[masks] format, for DTB modes offset is not needed, and a special size -1 means the part should fill all the remaining space

Starting from this, you can use dedit mode to remove partitions you don't want (^ + name selects a partition by its name, ? deletes the partition):

ampart /dev/mmcblk2 --mode dedit ^recovery? ^dtbo? ^cri_data? ^param? ^boot? ^rsv? ^metadata? ^vbmeta? ^tee? ^vendor? ^odm? ^system? ^product? ^cache? --dry-run

This brings the EPT to like this, where logo and misc is kept, so during booting you could still see the logo, and some misc things set by Android in misc partition could still be effective:

===================================================================================
ID| name            |          offset|(   human)|            size|(   human)| masks
-----------------------------------------------------------------------------------
 0: bootloader                      0 (   0.00B)           400000 (   4.00M)      0
    (GAP)                                                 2000000 (  32.00M)
 1: reserved                  2400000 (  36.00M)          4000000 (  64.00M)      0
    (GAP)                                                  800000 (   8.00M)
 2: cache                     6c00000 ( 108.00M)                0 (   0.00B)      0
    (GAP)                                                  800000 (   8.00M)
 3: env                       7400000 ( 116.00M)           800000 (   8.00M)      0
    (GAP)                                                  800000 (   8.00M)
 4: logo                      8400000 ( 132.00M)           800000 (   8.00M)      1
    (GAP)                                                  800000 (   8.00M)
 5: misc                      9400000 ( 148.00M)           800000 (   8.00M)      1
    (GAP)                                                  800000 (   8.00M)
 6: data                      a400000 ( 164.00M)       1d14800000 ( 116.32G)      4
===================================================================================

So you can have 4M-36M, 100M-116M, 117M-132M, 140M-148M, 156M-end to use, armbian-install could be adapted like this:

   elif [[ "${boxid}" -eq "xxx" ]]; then
       BLANK1="156"
       BOOT="256"
       BLANK2="0"

But if you just want say goodbye to all the stuffs not related to the Linux distro you're going to use, you can go further to delete even the logo and misc partitions:

ampart /dev/mmcblk2 --mode dedit ^logo? ^misc?

Then this will get you a partition table mentioned at the beginning.

And if we take a snapshot in this situation, the snapshot would be like:

data::-1:4

Since there's only a single data partition in the DTB partitions, whose size should be automatically generated to fill up all the remaining eMMC space, and its masks is 4 (not related to Linux booting, but for Android, 4 means its a data partition that could be wiped)

Additional notes:

ampart needs to edit the stock Android DTB stored at 36M+4M = 40M on the eMMC, as Amlogic's stock u-boot would generate EPT on each boot from the partitions node in that DTB, with which it will compare another EPT stored at 36M on eMMC, and update the on-eMMC one if the two are different, and the u-boot would consider the on-eMMC one broken and update accroding to the one generated from DTB. So the stock Android DTB must be modifiable.
ampart would try its best to preserve the DTB, and would remove the partitions node in the DTB as the last resort in direct EPT-modification modes (eclone, ecreate, eedit), for some boxes the u-boot may refuse to work if it found DTB "broken" like this, so DTB-modification modes (dclone, dedit) are always preferred
ampart supports plain DTB (the generic FDT format), Amlogic's multi DTB, and gzipped version of these two formats. However, it does not support encrypted DTB which is generated with each vendor's unique private key. The following devices are known to have encrypted DTB so ampart can't be used on them:
- Phicomm N1
- Xiaomi devices with >=Android 8 firmware
For DTB-modification modes (dclone, dedit), ampart would stick to the way Amlogic generates EPT to update the EPT. This has the following limitations:
- The reserved partition would always has a 32M gap before it, the bootloader partition would always be placed at 0
- The other partitions would always has a 8M before them
- The partitions must be placed incrementally, no one could either overlap with each other, or placed before another partition
- These are defined in Amlogic's u-boot so you have to stick to this way. Only specific vendors (I only know Xiaomi) would change this so their partition layout would be more efficcient.
Some newer devices requires more partitions to boot. I have yet only found HK1 Rbox X4 (with S905X4, not supported here) which will refuse to boot if either boot_a, vbmeta_a, vbmeta_system_a is missing, so the argument used in dclone mode should be changed to like this:
```
boot_a::64M:1 vbmeta_a::2M:1 vbmeta_system_a::2M:1 data::-1:4
```
You'll need to try and error all different partitions by yourself one by one if your box is like this. The good news is that you can unpack an Android burning image and edit its DTB with ampart directly with DTB-modification modes, and remove unneccessary big Android data partitions, and then re-pack to get a ~4M burning image, so you can recover the box to a working state very fast since you don't need Android stuffs.

The text was updated successfully, but these errors were encountered:

ophub · 2022-11-10T08:55:59Z

I want to put ampart in armbian firmware (/usr/bin/ampart)

This feature is added by default in armbian-install, such as this

try_ampart() {
    ampart /dev/mmcblk2 --mode dclone data::-1:4 >/dev/null 2>&1

    IFS=$'\n'
    dtb_partition_snapshots=($(ampart /dev/mmcblk2 --mode dsnapshot 2>/dev/null))
    unset IFS

    dtb_decimal_snapshot=${dtb_partition_snapshots[0]}

    dtb_partitions=(${dtb_decimal_snapshot})

    [[ "${dtb_partitions[@]}" == "data::-1:4" ]] && echo "CLI write EPT: Write successful"
}


create_partition() {
    #.....

    if [[ "${dtb_partitions[@]}" == "data::-1:4" ]]; then
        BLANK1="117"
        BOOT="256"
        BLANK2="0"
 

    #....

}

If the partition is successful, its value is a fixed value: data::-1:4
right?

7Ji · 2022-11-10T08:57:05Z

@ophub

[[ ${dtb_partitions} == "data::-1:4"  ]] && echo "CLI write EPT: Write successful"

This is not correct due to the Bash syntax for array

For bash arrays, if you just use its name, that'll be only the first item, to refer to the whole array, use a [@], the line should be changed to like this:

[[ "${dtb_partitions[@]}" == "data::-1:4"  ]] && echo "CLI write EPT: Write successful"

An example script to only do the adjustment once:

# $1: the eMMC device, or the block device for reserved partition, or a DTB block device or the dumpped image of these three devices
SNAPSHOT='data::-1:4'
LOG=$(ampart "$1" --mode dsnapshot 2>/dev/null)
if (( $? )); then
  echo 'Failed to get DTB snapshot!'
  exit 1
fi
readarray -d $'\n' -t DTB_SNAPSHOTS <<< "${LOG}"
# 0 is decimal, 1 is hex, 2 is human-readable
echo "Please use the following snapshot to restore the DTB partitions if anything goes wrong:"
echo "${DTB_SNAPSHOTS[2]}"
# Note an addtional ' ' here, it's implicitly added after each partition. The comparasion could also be done after the new array PARTITIONS is created, but that's not neccessary if we can do it here right?
if [ "${DTB_SNAPSHOTS[0]}" == "${SNAPSHOT} " ]; then
  echo 'DTB partitions layout has already been modified, no need to update it'
  exit 0
fi
readarray -d ' ' -t PARTITIONS < <(printf '%s' "${DTB_SNAPSHOTS[0]}")
if ! (( ${#PARTITIONS[@]} )); then
  echo 'No partitions in DTB, this is impossible, did you use ampart to modify the DTB yourself?'
  exit 2
fi
echo "Existing partitions in DTB:"
for i in $(seq 0 $(("${#PARTITIONS[@]}"-1)) ); do
  readarray -d ':' -t PARTINFO < <(printf '%s' "${PARTITIONS[$i]}")
  echo "Partition $i: ${PARTINFO[0]}, size ${PARTINFO[2]}, masks ${PARTINFO[3]}"
done
ampart "$1" --mode dclone ${SNAPSHOT} 2>/dev/null
if (( $? )); then
  echo 'Failed to adjust DTB partitions!'
  exit 3
fi
echo 'Successfully adjusted DTB partitions'

Two continous runs yield the following output:

[nomad7ji@laptop7ji images]$ sh ../scripts/dclone_once.sh x3_emmc.img 
Please use the following snapshot to restore the DTB partitions if anything goes wrong:
logo::8M:1 recovery::24M:1 misc::8M:1 dtbo::8M:1 cri_data::8M:2 param::16M:2 boot::16M:1 rsv::16M:1 metadata::16M:1 vbmeta::2M:1 tee::32M:1 vendor::320M:1 odm::128M:1 system::1856M:1 product::128M:1 cache::1120M:2 data::-1:4 
Existing partitions in DTB:
Partition 0: logo, size 8388608, masks 1
Partition 1: recovery, size 25165824, masks 1
Partition 2: misc, size 8388608, masks 1
Partition 3: dtbo, size 8388608, masks 1
Partition 4: cri_data, size 8388608, masks 2
Partition 5: param, size 16777216, masks 2
Partition 6: boot, size 16777216, masks 1
Partition 7: rsv, size 16777216, masks 1
Partition 8: metadata, size 16777216, masks 1
Partition 9: vbmeta, size 2097152, masks 1
Partition 10: tee, size 33554432, masks 1
Partition 11: vendor, size 335544320, masks 1
Partition 12: odm, size 134217728, masks 1
Partition 13: system, size 1946157056, masks 1
Partition 14: product, size 134217728, masks 1
Partition 15: cache, size 1174405120, masks 2
Partition 16: data, size -1, masks 4
Successfully adjusted DTB partitions
[nomad7ji@laptop7ji images]$ sh ../scripts/dclone_once.sh x3_emmc.img 
Please use the following snapshot to restore the DTB partitions if anything goes wrong:
data::-1:4 
DTB partitions layout has already been modified, no need to update it

ophub · 2022-11-10T09:01:48Z

Ok, I have corrected the above error, you can check whether the mode I choose is the best (compatibility, applicability), is this usage correct?

I put your ampart in the Armbian firmware, it comes with it by default. Does it comply with your license agreement?

7Ji · 2022-11-10T09:09:44Z

Yes, something could be changed but they're just style preferences (i.e. data::-1:4 as a constant variable instead of two occurances, using readarray instead of setting IFS to avoid the cases where it is set in a larger scope yet unset here)

Does it comply with your license agreement?

Yes, I'm open with its inclusion in other open-source projects. But would it be a pre-built binary or being built during firmware packing? Because I'm lazy with releases and the aarch64-static one is pretty big, so using the pre-built static aarch64 one would mean it would be (probably) old and big. You could build it yourself on your Armbian distro natively and use the dynamically linked one built by yourself, or with a cross toolchain. I don't limit the form how ampart come in other distros.

ophub · 2022-11-10T09:20:51Z

try_ampart() {
    ampart /dev/mmcblk2 --mode dclone data::-1:4 >/dev/null 2>&1

    SNAPSHOT='data::-1:4'
    LOG=$(ampart /dev/mmcblk2 --mode dsnapshot 2>/dev/null)
    readarray -d $'\n' -t DTB_SNAPSHOTS <<<"${LOG}"
    [[ "${DTB_SNAPSHOTS[@]}" == "${SNAPSHOT}" ]] && echo "CLI write EPT: Write successful"
}

create_partition() {
    #.....

    if [[ "${dtb_partitions[@]}" == "${SNAPSHOT}" ]]; then
        BLANK1="117"
        BOOT="256"
        BLANK2="0"
 

    #....

}

7Ji · 2022-11-10T11:18:26Z

I've adjusted the snapshot modes and now the last partition in snapshots won't have extra trailing space, please use the newest release:
https://github.com/7Ji/ampart/releases/tag/v1.1

"${DTB_SNAPSHOTS[@]}" == "${SNAPSHOT}"

Before the update, the comparasion would be false since there will be an extra space after the last partition. The comparasion would be "data::-1:4 " == "data::-1:4" and it would be false as a result.

With commit 7Ji/ampart@bd4409b, the last partition in a snapshot will not have trailing space anymore, the logic is now true and more intuitive.

About the codes,

something could be changed but they're just style preferences (i.e. data::-1:4 as a constant variable

I was refering to a layout like this:

#...
try_ampart() {
    # This is declared once and used multiple time, so you and the users can just change it once in the future if you want a different layout
    local snapshot_expected='data::-1:4' 

    # Uses your variable install_emmc to avoid wrting to wrong disk. Note there's no quote around ${snapshot_expected}, because there could be multiple partitions in the snapshot, and they would need to be splitted on space
    ampart "${install_emmc}" --mode dclone ${snapshot_expected} &>/dev/null

    if (( $? )); then
        # Handle the situation where ampart fails to change the layout
        return 1
    fi
    
    local log=$(ampart "${install_emmc}" --mode dsnapshot 2>/dev/null)

    if (( $? )); then
        # Handle the situation where ampart fails to read the new partitions
        return 2
    fi

    local snapshots
    readarray -d $'\n' -t snapshots << "${log}"

    # Only one of the snapshots need to be compared, 0 here refers to the decimal one, if you would like to enable users to change stuffs, i.e. declare there layout as snapshot_expected, 2 for the human-readable version could probably be better. 0 and 1 are better if you need to calculate on these values hence they're called script-friendly
    if [[ "${snapshots[0]}" == "${snapshot_expected}" ]]; then
        # Successfully written the partitions
        echo "CLI write EPT: Write successful"
        return 0
    else
        # Result partitions different, maybe should handle it
        return 3

    fi
}

create_partition() {
    #.....

    try_ampart
    r_ampart=$?

    #....

    if [[ $r_ampart == 0 ]]; then
        BLANK1="117"
        BOOT="256"
        BLANK2="0"
 

    #....

}

ophub · 2022-11-10T12:10:13Z

ampart "/dev/mmcblk2" --mode dclone data::-1:4 &>/dev/null
ampart "mmcblk2" --mode dclone data::-1:4 &>/dev/null

The format of /dev/mmcblk2 should be correct, right?

7Ji · 2022-11-10T12:25:15Z

Yes, the first positional argument given to ampart would be parsed as the target, which would be opened for I/O operation, it could be any "file" (in the Linux sense) as long as it could be opened for seek, read and write. That means to use ampart to edit an eMMC drive whose block device presents as /dev/mmcblk2, you should use /dev/mmcblk2 as the target (the second one mmcblk2 would also be correct if you're operating inside /dev, or there's an dumped image file in the current working directory whose name is mmcblk2).

This also means, for test and development, you can give the path of an eMMC dump you got with dd to ampart, in that case it'll operate on the file and not directly on the drive, so you can confirm if some arguments are correct. ampart has also another non-positional argument --dry-run which will disable the writing completely, so you can check the logs to confirm whether your arguments result in a partition layout you want, before actually using them

In the following command:

ampart "/dev/mmcblk2" --mode dclone data::-1:4

/dev/mmcblk2 is the first positional argument, --mode is the a non-positional argument, dclone is the argument for --mode, data::-1:4 is the second positional argument, this command could also be written as any of the following format:

ampart /dev/mmcblk2 data::-1:4 --mode dclone
ampart --mode dclone /dev/mmcblk2 data::-1:4

As long as the first positional argument is the target, and the other positional arguments are partitions, the command is correct

ophub · 2022-11-10T14:03:53Z

1e4c566

Added ampart as default partition tool

If you need to keep the original Android partition table, you can specify it in the second parameter of armbian-install: armbian-install no no

ophub · 2022-11-11T04:39:32Z

你的ampart工具在分区日志里输出的分区表很直观。
能否开发一个工具，可以一键显示当前安卓系统分区表。这样方便一些开发工作。
我之前使用的安卓分区没认识到各分区间隔之间8mb的大小，可能部分分区大小设置有错误。想准确校正一下相关大小，这样锁了DTB的盒子，不能重新调整分区表时，可以更加准确地设置可用分区。

7Ji · 2022-11-11T07:32:33Z

I've added a demo script that can automatically identify the eMMC block device and report EPT. The result table has more columns (end of each partition).
增加了一个能自动识别eMMC块设备并汇报EPT的演示脚本。结果表比ampart本身汇报EPT时多了两列（分区的结尾）

Documentation: https://github.com/7Ji/ampart/blob/ampart/doc/scripting-what-is-my-ept.md
Script: https://github.com/7Ji/ampart/blob/ampart/scripts/what-is-my-ept.py

Log:

[root@bestv7Ji ~]# /tmp/what-is-my-ept.py 
Automatically chosen /dev/mmcblk2 as eMMC
5 partitions in the EPT
======================================================================================================
id|name           |            start| (human)|            size| (human)|            end| (human)|masks
======================================================================================================
 0 bootloader                      0    0.00B           400000    4.00M           400000    4.00M  0
   (gap)                                               2000000   32.00M
 1 reserved                  2400000   36.00M          4000000   64.00M          6400000  100.00M  0
   (gap)                                                800000    8.00M
 2 cache                     6c00000  108.00M                0    0.00B          6c00000  108.00M  0
   (gap)                                                800000    8.00M
 3 env                       7400000  116.00M           800000    8.00M          7c00000  124.00M  0
   (gap)                                                800000    8.00M
 4 data                      8400000  132.00M        1c9c00000    7.15G        1d2000000    7.28G  4
======================================================================================================

ophub · 2022-11-11T10:02:56Z

太棒了，我学习学习，做个使用说明

hhalibo · 2022-11-13T11:59:26Z

@7Ji Good job！Wonderful tool for armbian-AML，especially for the home-assistant user like me！THANKS！

7Ji · 2023-01-08T04:26:12Z

https://7ji.github.io/crack/2023/01/08/decrypt-aml-dtb.html

An updated note, it's possible to extract plain DTB from running Android on a box with encrypted DTB, and then it's possible to re-partition eMMC with ampart.
更新，对于有加密DTB的盒子，可以从正在运行的安卓提取明文的DTB，然后就可以用ampart给eMMC分区了

livelier · 2023-01-08T04:44:58Z

7大威武 👍

ophub added documentation Improvements or additions to documentation support This need is supported essence Recommended essence posts labels Nov 10, 2022

ophub closed this as completed Nov 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Utilizing more space for eMMC installation with ampart / 使用ampart来在eMMC安装中使用更多空间 #689

Utilizing more space for eMMC installation with ampart / 使用ampart来在eMMC安装中使用更多空间 #689

7Ji commented Nov 10, 2022 •

edited

01. Resize partition

02. If the adjustment is successful, it will output information:

03. Modify the [ armbian-install ] of the a script, set your own device id to use the full partition

ophub commented Nov 10, 2022 •

edited

7Ji commented Nov 10, 2022

ophub commented Nov 10, 2022 •

edited

7Ji commented Nov 10, 2022

ophub commented Nov 10, 2022 •

edited

7Ji commented Nov 10, 2022 •

edited

ophub commented Nov 10, 2022 •

edited

7Ji commented Nov 10, 2022 •

edited

ophub commented Nov 10, 2022

ophub commented Nov 11, 2022 •

edited

7Ji commented Nov 11, 2022

ophub commented Nov 11, 2022

hhalibo commented Nov 13, 2022

7Ji commented Jan 8, 2023

livelier commented Jan 8, 2023

Utilizing more space for eMMC installation with ampart / 使用ampart来在eMMC安装中使用更多空间 #689

Utilizing more space for eMMC installation with ampart / 使用ampart来在eMMC安装中使用更多空间 #689

Comments

7Ji commented Nov 10, 2022 • edited

01. Resize partition

02. If the adjustment is successful, it will output information:

03. Modify the [ armbian-install ] of the a script, set your own device id to use the full partition

ophub commented Nov 10, 2022 • edited

7Ji commented Nov 10, 2022

ophub commented Nov 10, 2022 • edited

7Ji commented Nov 10, 2022

ophub commented Nov 10, 2022 • edited

7Ji commented Nov 10, 2022 • edited

ophub commented Nov 10, 2022 • edited

7Ji commented Nov 10, 2022 • edited

ophub commented Nov 10, 2022

ophub commented Nov 11, 2022 • edited

7Ji commented Nov 11, 2022

ophub commented Nov 11, 2022

hhalibo commented Nov 13, 2022

7Ji commented Jan 8, 2023

livelier commented Jan 8, 2023

7Ji commented Nov 10, 2022 •

edited

ophub commented Nov 10, 2022 •

edited

ophub commented Nov 10, 2022 •

edited

ophub commented Nov 10, 2022 •

edited

7Ji commented Nov 10, 2022 •

edited

ophub commented Nov 10, 2022 •

edited

7Ji commented Nov 10, 2022 •

edited

ophub commented Nov 11, 2022 •

edited