Investigate "not properly aligned for best performance" #102

Closed
dagwieers opened this Issue Jun 7, 2012 · 17 comments

Comments

Projects
None yet
6 participants
@dagwieers
Member

dagwieers commented Jun 7, 2012

When doing a rear restore parted complains about the fact that partitions are not properly aligned. This might have a performance impact on some hardware and therefor we need to make sure if parted is correct in its claims and whether we should improve our logic (in those cases where that's possible depending on parted version).

In my case I have seen this:

 +++ echo -e 'Creating partitions for disk /dev/sda (msdos)'
 +++ parted -s /dev/sda mklabel msdos
 +++ parted -s /dev/sda mkpart primary 32768B 542836735B
 Warning: The resulting partition is not properly aligned for best performance.
 +++ parted -s /dev/sda set 1 boot on
 +++ parted -s /dev/sda mkpart primary 542838784B 240057409535B
 Warning: The resulting partition is not properly aligned for best performance.
 +++ parted -s /dev/sda set 2 lvm on

Looking at the offsets, the main question is whether the offsets starts at 0 or 1. Once we have determined what the correct start offset is, we can make sure the offsets used are aligned on 4k boundaries. Here is an example for the above boundaries:

[root@moria ~]# echo alignment: $((32768%4096)) $((542836735%4096))
alignment: 0 2047
[root@moria ~]# echo alignment: $((542838784%4096)) $((240057409535%4096))
alignment: 0 4095

So in the above case, either the start offset, or the stop offset are off-by-one. In essence, we should only be concerned by the start offset, but we could do the right thing anyway.

@dagwieers

This comment has been minimized.

Show comment
Hide comment
@dagwieers

dagwieers Jun 9, 2012

Member

Looking into this a bit closer, it's hard to know what properly aligned means. According to some fora, if the EBS (Erase Block Size) is unknown, using 512*1024 is a safe size to use. Looking at my offset:

[root@moria rear]# parted /dev/sda unit b print
Model: ATA Corsair Force GT (scsi)
Disk /dev/sda: 240057409536B
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start       End            Size           Type     File system  Flags
 1      32768B      542836735B     542803968B     primary  ext4         boot
 2      542838784B  240057409535B  239514570752B  primary               lvm

The first partition does not start at an erase block, but the second partition does:

[root@moria rear]# echo $((542838784%512*1024))
0

Still parted complained ?

Member

dagwieers commented Jun 9, 2012

Looking into this a bit closer, it's hard to know what properly aligned means. According to some fora, if the EBS (Erase Block Size) is unknown, using 512*1024 is a safe size to use. Looking at my offset:

[root@moria rear]# parted /dev/sda unit b print
Model: ATA Corsair Force GT (scsi)
Disk /dev/sda: 240057409536B
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start       End            Size           Type     File system  Flags
 1      32768B      542836735B     542803968B     primary  ext4         boot
 2      542838784B  240057409535B  239514570752B  primary               lvm

The first partition does not start at an erase block, but the second partition does:

[root@moria rear]# echo $((542838784%512*1024))
0

Still parted complained ?

@kpieth

This comment has been minimized.

Show comment
Hide comment
@kpieth

kpieth Oct 17, 2015

This is an important feature, because after a rear recovery you always have unaligned Partitions and slow systems.
Currently I always have to manually enter Starting Offsets before recovery. The starting Position of the first partition is on my Systems(HP Proliant, VMware, SSD Laptop) always Sector 2048 to get proper alignment. A good explanation in german can be found here:
https://www.thomas-krenn.com/de/wiki/Partition_Alignment

kpieth commented Oct 17, 2015

This is an important feature, because after a rear recovery you always have unaligned Partitions and slow systems.
Currently I always have to manually enter Starting Offsets before recovery. The starting Position of the first partition is on my Systems(HP Proliant, VMware, SSD Laptop) always Sector 2048 to get proper alignment. A good explanation in german can be found here:
https://www.thomas-krenn.com/de/wiki/Partition_Alignment

@schlomo

This comment has been minimized.

Show comment
Hide comment
@schlomo

schlomo Oct 17, 2015

Member

@kpieth yes you are very right. Unfortunately nobody figured out an automated way of doing this so far. Any ideas?

Member

schlomo commented Oct 17, 2015

@kpieth yes you are very right. Unfortunately nobody figured out an automated way of doing this so far. Any ideas?

@jsmeix

This comment has been minimized.

Show comment
Hide comment
@jsmeix

jsmeix Oct 19, 2015

Contributor

Only at a very quick first glance:

In the rear 1.17.2 sources I do not find any usage of "parted ... --align=optimal"

It seems regardless that in usr/share/rear/layout/prepare/GNU/Linux/10_include_partition_code.sh a FEATURE_PARTED_ALIGNMENT exists, parted alignment is nowhere used.

I am not at all a sufficient parted expert to understand if "parted ... --align=optimal" is actually the right solution here. The wording makes it look as if it does "the right thing" but I do not (yet) know if it actually does the right thing because http://www.gnu.org/software/parted/manual/parted.html is not really explanatory what actually happens for each of parted's alignment types none, cylinder, minimal and optimal.

At least when YaST installs an openSUSE or SLE system it calls "parted ... --align=optimal", see also my script at "Generic disaster recovery with the plain SUSE installation system" in https://en.opensuse.org/SDB:Disaster_Recovery how I call parted there (basicaly I copied it from how YaST calls parted, see the comments in my script).

Contributor

jsmeix commented Oct 19, 2015

Only at a very quick first glance:

In the rear 1.17.2 sources I do not find any usage of "parted ... --align=optimal"

It seems regardless that in usr/share/rear/layout/prepare/GNU/Linux/10_include_partition_code.sh a FEATURE_PARTED_ALIGNMENT exists, parted alignment is nowhere used.

I am not at all a sufficient parted expert to understand if "parted ... --align=optimal" is actually the right solution here. The wording makes it look as if it does "the right thing" but I do not (yet) know if it actually does the right thing because http://www.gnu.org/software/parted/manual/parted.html is not really explanatory what actually happens for each of parted's alignment types none, cylinder, minimal and optimal.

At least when YaST installs an openSUSE or SLE system it calls "parted ... --align=optimal", see also my script at "Generic disaster recovery with the plain SUSE installation system" in https://en.opensuse.org/SDB:Disaster_Recovery how I call parted there (basicaly I copied it from how YaST calls parted, see the comments in my script).

@jsmeix

This comment has been minimized.

Show comment
Hide comment
@jsmeix

jsmeix Oct 19, 2015

Contributor

A general question:

Because at least low-end flash devices (a.k.a. USB sticks) require alignment at 4MiB or even 8 MiB for not-too-bad performance (cf. my last comment at https://hackweek.suse.com/12/projects/23 ) I wonder if rear should by default align at 4MiB or even 8 MiB - at least when the partition size is much bigger than 4MiB or 8 MiB (e.g. for partitions bigger that 100 MB).

This way a badly aligned original system could become even faster after recovery. Is such a difference allowed or must rear recovery slavishly produce a byte-by-byte identical copy whenever possible?

Contributor

jsmeix commented Oct 19, 2015

A general question:

Because at least low-end flash devices (a.k.a. USB sticks) require alignment at 4MiB or even 8 MiB for not-too-bad performance (cf. my last comment at https://hackweek.suse.com/12/projects/23 ) I wonder if rear should by default align at 4MiB or even 8 MiB - at least when the partition size is much bigger than 4MiB or 8 MiB (e.g. for partitions bigger that 100 MB).

This way a badly aligned original system could become even faster after recovery. Is such a difference allowed or must rear recovery slavishly produce a byte-by-byte identical copy whenever possible?

@schlomo

This comment has been minimized.

Show comment
Hide comment
@schlomo

schlomo Oct 19, 2015

Member

Wild guess: I am not sure if --align=optimal will have any effect as long as we give partedexact Byte ranges as input. After all it would have to change our precise input to make the alignment. Maybe part of the problem is also to convert the math to work in MB so that partedwill have some leeway how to interpret the MB numbers.

Member

schlomo commented Oct 19, 2015

Wild guess: I am not sure if --align=optimal will have any effect as long as we give partedexact Byte ranges as input. After all it would have to change our precise input to make the alignment. Maybe part of the problem is also to convert the math to work in MB so that partedwill have some leeway how to interpret the MB numbers.

@jsmeix

This comment has been minimized.

Show comment
Hide comment
@jsmeix

jsmeix Oct 19, 2015

Contributor

Yes, that exact byte ranges as input for parted is what I question if that is really reasonable (at least reasonable by default) to be done by rear in my above comment regarding using 4MiB or even 8 MiB alignment by default (for for sufficiently big partitions).

In other words: I wonder if rear should by default round exact byte values to 4MiB or even 8 MiB chunks?

An obvious direct drawback is that a partition could become several MiB smaller than it was before.

As an extreme example assume a 2020 MiB disk consists of 10 partitions each with 202 MiB. Each is rounded to 4MiB resulting the first 9 partitions rounded to 204 MiB. As a consequence for the last partition only 184 MiB = 2020 MiB - ( 9 * 204 MiB ) are available space on the disk so that after recovery the 10. partition becomes 18 MiB = 9 * 2 MiB smaller than it was originally (i.e. about 10% smaller in this example).

With GPT partitioning and 110 such 202 MiB partitions on a 22220 MiB disk it is left to the reader to calculate that there would be no space left on the device for the last partition.

Contributor

jsmeix commented Oct 19, 2015

Yes, that exact byte ranges as input for parted is what I question if that is really reasonable (at least reasonable by default) to be done by rear in my above comment regarding using 4MiB or even 8 MiB alignment by default (for for sufficiently big partitions).

In other words: I wonder if rear should by default round exact byte values to 4MiB or even 8 MiB chunks?

An obvious direct drawback is that a partition could become several MiB smaller than it was before.

As an extreme example assume a 2020 MiB disk consists of 10 partitions each with 202 MiB. Each is rounded to 4MiB resulting the first 9 partitions rounded to 204 MiB. As a consequence for the last partition only 184 MiB = 2020 MiB - ( 9 * 204 MiB ) are available space on the disk so that after recovery the 10. partition becomes 18 MiB = 9 * 2 MiB smaller than it was originally (i.e. about 10% smaller in this example).

With GPT partitioning and 110 such 202 MiB partitions on a 22220 MiB disk it is left to the reader to calculate that there would be no space left on the device for the last partition.

@jsmeix

This comment has been minimized.

Show comment
Hide comment
@jsmeix

jsmeix Oct 19, 2015

Contributor

@kpieth regarding
https://www.thomas-krenn.com/de/wiki/Partition_Alignment
that reads (excerpt)

4 KiB / 8 KiB Pages der SSDs

I do not agree from my current point of understanding.

For flash strorage the physical block size for read/write operations is usually something like 4MiB or 8MiB (mega bytes not kilo bytes!) cf. my last comment in https://hackweek.suse.com/12/projects/23

I think SSDs usually have sufficiently intelligent controllers with sufficiently huge caches built in so that small reads or writes could be sufficiently well buffered - in particular consecutive small reads or writes are combined by the controller into big reads or writes that mach the physical block size of the actual storage hardware.

I think in the end for a sufficiently well made SSD it should not matter too much how it is accessed or aligned.

In contrast I think for cheap flash-based storagae it does matter in what chunks and with what alignment it is accessed.

But I am not at all a sufficient expert in this area to make really authoritative statements.

I only think that 4 KiB alignment which should be the right one for spinning traditional harddisks could be totally insufficient for cheap flash based storage where 4 MiB or even 8 MiB would have to be used.

This is the reason why I am thinking about using 4 MiB or even 8 MiB alingnment by default in rear for sufficiently big partitions (i.e. where rounding differences should not matter in reasonable scenarios).

Contributor

jsmeix commented Oct 19, 2015

@kpieth regarding
https://www.thomas-krenn.com/de/wiki/Partition_Alignment
that reads (excerpt)

4 KiB / 8 KiB Pages der SSDs

I do not agree from my current point of understanding.

For flash strorage the physical block size for read/write operations is usually something like 4MiB or 8MiB (mega bytes not kilo bytes!) cf. my last comment in https://hackweek.suse.com/12/projects/23

I think SSDs usually have sufficiently intelligent controllers with sufficiently huge caches built in so that small reads or writes could be sufficiently well buffered - in particular consecutive small reads or writes are combined by the controller into big reads or writes that mach the physical block size of the actual storage hardware.

I think in the end for a sufficiently well made SSD it should not matter too much how it is accessed or aligned.

In contrast I think for cheap flash-based storagae it does matter in what chunks and with what alignment it is accessed.

But I am not at all a sufficient expert in this area to make really authoritative statements.

I only think that 4 KiB alignment which should be the right one for spinning traditional harddisks could be totally insufficient for cheap flash based storage where 4 MiB or even 8 MiB would have to be used.

This is the reason why I am thinking about using 4 MiB or even 8 MiB alingnment by default in rear for sufficiently big partitions (i.e. where rounding differences should not matter in reasonable scenarios).

@jsmeix

This comment has been minimized.

Show comment
Hide comment
@jsmeix

jsmeix Oct 20, 2015

Contributor

Wait!
Going back to square one:

I verified that with "rear recover" (using rear 1.17.2)
one gets a byte-by-byte identical copy of the partitioning
of the original disk:

My original system disk:

# parted -s /dev/sda unit B print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 26843545600B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 
Number  Start         End           Size          Type     File system     Flags
 1      1048576B      1570766847B   1569718272B   primary  linux-swap(v1)  type=82
 2      1570766848B   14459863039B  12889096192B  primary  btrfs           boot, type=83
 3      14459863040B  26843545599B  12383682560B  primary  xfs             type=83

My recovered system disk on a identical second machine:

# parted -s /dev/sda unit B print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 26843545600B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 
Number  Start         End           Size          Type     File system     Flags
 1      1048576B      1570766847B   1569718272B   primary  linux-swap(v1)  type=83
 2      1570766848B   14459863039B  12889096192B  primary  btrfs           boot, type=83
 3      14459863040B  26843545599B  12383682560B  primary  xfs             type=83

This means when after "rear recover" the partitions are badly aligned,
they must have been already badly aligned on the original system.

I think it does not belong to rear to somehow try to fix badly aligned
partitions.

Or do I misunderstand something so that it can happen
that well aligned partitions on the original system become
badly aligned partitions after "rear recover"?

@kpieth
please provide the output of

# parted -s /dev/sdX unit B print

(replaxe 'X' with what is appropriate for your harddisk)
both on your original system and after "rear recover".

Contributor

jsmeix commented Oct 20, 2015

Wait!
Going back to square one:

I verified that with "rear recover" (using rear 1.17.2)
one gets a byte-by-byte identical copy of the partitioning
of the original disk:

My original system disk:

# parted -s /dev/sda unit B print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 26843545600B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 
Number  Start         End           Size          Type     File system     Flags
 1      1048576B      1570766847B   1569718272B   primary  linux-swap(v1)  type=82
 2      1570766848B   14459863039B  12889096192B  primary  btrfs           boot, type=83
 3      14459863040B  26843545599B  12383682560B  primary  xfs             type=83

My recovered system disk on a identical second machine:

# parted -s /dev/sda unit B print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 26843545600B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 
Number  Start         End           Size          Type     File system     Flags
 1      1048576B      1570766847B   1569718272B   primary  linux-swap(v1)  type=83
 2      1570766848B   14459863039B  12889096192B  primary  btrfs           boot, type=83
 3      14459863040B  26843545599B  12383682560B  primary  xfs             type=83

This means when after "rear recover" the partitions are badly aligned,
they must have been already badly aligned on the original system.

I think it does not belong to rear to somehow try to fix badly aligned
partitions.

Or do I misunderstand something so that it can happen
that well aligned partitions on the original system become
badly aligned partitions after "rear recover"?

@kpieth
please provide the output of

# parted -s /dev/sdX unit B print

(replaxe 'X' with what is appropriate for your harddisk)
both on your original system and after "rear recover".

@jhoekx

This comment has been minimized.

Show comment
Hide comment
@jhoekx

jhoekx Oct 20, 2015

Member

When I was working on the partitioning code, the goal was to reuse the exact same offsets if no resizing had to be done. As soon as we enter migration mode, the code tried to align on boundaries.

It would be interesting to know if we are in migration mode or not?

Maybe the choice of boundaries is not optimal?

Member

jhoekx commented Oct 20, 2015

When I was working on the partitioning code, the goal was to reuse the exact same offsets if no resizing had to be done. As soon as we enter migration mode, the code tried to align on boundaries.

It would be interesting to know if we are in migration mode or not?

Maybe the choice of boundaries is not optimal?

@kpieth

This comment has been minimized.

Show comment
Hide comment
@kpieth

kpieth Oct 23, 2015

Yes we are talking about migration mode. A well aligned partition gets bad aligned after rear recovery. If no resizing is done, everything is at it was. We use rear for installing new machines(VM and Hardware). My machines usually run virtualized and so have several layers of storage under it. Alignment really has a performance impact.

kpieth commented Oct 23, 2015

Yes we are talking about migration mode. A well aligned partition gets bad aligned after rear recovery. If no resizing is done, everything is at it was. We use rear for installing new machines(VM and Hardware). My machines usually run virtualized and so have several layers of storage under it. Alignment really has a performance impact.

@gdha

This comment has been minimized.

Show comment
Hide comment
@gdha

gdha Sep 7, 2016

Member

added it to the sponsor list - close it

Member

gdha commented Sep 7, 2016

added it to the sponsor list - close it

@jsmeix

This comment has been minimized.

Show comment
Hide comment
@jsmeix

jsmeix Feb 13, 2018

Contributor

Right now the following is mainly only an offhanded guess
but perhaps I may have found the root cause of this issue:

With bigger replacement disk size one gets partitions recreated
with some automatically resized partitions via
usr/share/rear/layout/prepare/default/400_autoresize_disks.sh
which evenly distribute the size changes on byte values so that
the automatically resized partitions can be arbitrarily badly aligned.

Later some partition alignment happens in
usr/share/rear/layout/prepare/GNU/Linux/100_include_partition_code.sh
where my offhanded guess is the alignment is something like 1 MiB
because of the '/ 1024 / 1024' in the code
but perhaps one may even get only 1 Byte as alignment
in case of FEATURE_PARTED_ANYUNIT which may finally
explain the root cause behind this issue.

Summary from my current point of view:

With same replacement disk size one gets partitions recreated
at the exact same byte values as they have been on the original system
which is the right behaviour because ReaR is first and foremost meant
to recreate a system as much as possible exactly as it was before.

In contrast with bigger replacement disk size one may get
automatically resized partitions that can be arbitrarily badly aligned.

I think even with bigger replacement disk size the default should also be
to get partitions recreated at the exact same byte values
as they have been on the original system so that by default one gets
the system recreated as much as possible exactly as it was before.

Optionally - only after a user confirmation dialog - with bigger replacement disk size
one could get automatically resized partitions but then with a specific
alignment value PARTITION_ALIGN_BLOCK_SIZE that is by default
the same as USB_PARTITION_ALIGN_BLOCK_SIZE i.e. 8 MiB
to create things reasonably right out of the box also on SSDs, cf.
#1201

Contributor

jsmeix commented Feb 13, 2018

Right now the following is mainly only an offhanded guess
but perhaps I may have found the root cause of this issue:

With bigger replacement disk size one gets partitions recreated
with some automatically resized partitions via
usr/share/rear/layout/prepare/default/400_autoresize_disks.sh
which evenly distribute the size changes on byte values so that
the automatically resized partitions can be arbitrarily badly aligned.

Later some partition alignment happens in
usr/share/rear/layout/prepare/GNU/Linux/100_include_partition_code.sh
where my offhanded guess is the alignment is something like 1 MiB
because of the '/ 1024 / 1024' in the code
but perhaps one may even get only 1 Byte as alignment
in case of FEATURE_PARTED_ANYUNIT which may finally
explain the root cause behind this issue.

Summary from my current point of view:

With same replacement disk size one gets partitions recreated
at the exact same byte values as they have been on the original system
which is the right behaviour because ReaR is first and foremost meant
to recreate a system as much as possible exactly as it was before.

In contrast with bigger replacement disk size one may get
automatically resized partitions that can be arbitrarily badly aligned.

I think even with bigger replacement disk size the default should also be
to get partitions recreated at the exact same byte values
as they have been on the original system so that by default one gets
the system recreated as much as possible exactly as it was before.

Optionally - only after a user confirmation dialog - with bigger replacement disk size
one could get automatically resized partitions but then with a specific
alignment value PARTITION_ALIGN_BLOCK_SIZE that is by default
the same as USB_PARTITION_ALIGN_BLOCK_SIZE i.e. 8 MiB
to create things reasonably right out of the box also on SSDs, cf.
#1201

@jsmeix

This comment has been minimized.

Show comment
Hide comment
@jsmeix

jsmeix Feb 16, 2018

Contributor

In my current #1733
I do not implement any PARTITION_ALIGN_BLOCK_SIZE support
because I think this is not needed when only the end value
of the last partition on each disk may get changed
(I use a hardcoded 1 MiB alignment for the partition end values)
for details see #1733

Contributor

jsmeix commented Feb 16, 2018

In my current #1733
I do not implement any PARTITION_ALIGN_BLOCK_SIZE support
because I think this is not needed when only the end value
of the last partition on each disk may get changed
(I use a hardcoded 1 MiB alignment for the partition end values)
for details see #1733

@jsmeix

This comment has been minimized.

Show comment
Hide comment
@jsmeix

jsmeix Feb 21, 2018

Contributor

Only FYI:

Right now it happened to me that I got such a parted
"Warning: The resulting partition is not properly aligned for best performance.".

In my case it happened this way:

# parted -s /dev/sdb unit MiB print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sdb: 2048MiB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number  Start    End      Size    Type      File system     Flags
 1      8.00MiB  808MiB   800MiB  primary   ext2            type=83
 2      808MiB   1208MiB  400MiB  primary   linux-swap(v1)  type=82
 3      1208MiB  1708MiB  500MiB  extended                  lba, type=0f

# parted -s -a optimal /dev/sdb unit MiB mkpart logical 1208 1408
Warning: The resulting partition is not properly aligned for best performance.

# parted -s /dev/sdb unit MiB print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sdb: 2048MiB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number  Start    End      Size    Type      File system     Flags
 1      8.00MiB  808MiB   800MiB  primary   ext2            type=83
 2      808MiB   1208MiB  400MiB  primary   linux-swap(v1)  type=82
 3      1208MiB  1708MiB  500MiB  extended                  lba, type=0f
 5      1208MiB  1408MiB  200MiB  logical                   type=83

# parted -s /dev/sdb unit B print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sdb: 2147483648B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number  Start        End          Size        Type      File system     Flags
 1      8388608B     847249407B   838860800B  primary   ext2            type=83
 2      847249408B   1266679807B  419430400B  primary   linux-swap(v1)  type=82
 3      1266679808B  1790967807B  524288000B  extended                  lba, type=0f
 5      1266680320B  1476395519B  209715200B  logical                   type=83

# echo '1208 * 1024 * 1024' | bc -l
1266679808

# echo '1266680320 / 1024 / 1024' | bc -l
1208.00048828125000000000

# echo '1266680320 - 1266679808' | bc -l
512

I had specified the logical partition start with 1 MiB unit
at the same point where the extended partition starts
but parted knows that the first sector (i.e. the first 512 bytes)
of the extended partition cannot be used by the logical partition
so that parted automatically moved the actual logical partition start
by only 512 bytes which results an actual logical partition start point
that is "not properly aligned for best performance"
(whatever that exactly means in parted's opinion - see next paragraph)
and - as far as I know - there should be an at least 63 sectors gap
(i.e. at least 63 * 512 bytes = 32256 bytes) gap between the
extended partition start and the first logical partition start.

I think when parted can make such a "Warning" message
cf. http://blog.schlomo.schapiro.org/2015/04/warning-is-waste-of-my-time.html
parted must have some built-in "knowledge" what would be
"properly aligned for best performance" and then I wonder why
parted does not automatically align it "properly for best performance"
regardless that I had explicitly called parted with -a optimal
i.e. what the heck is parted's alignment type optimal meant for?

When I manually specify the logical partition start with 1 MiB unit
at one MiB more from where the extended partition starts it works:

# parted -s /dev/sdb rm 5

# parted -s /dev/sdb unit MiB mkpart logical 1209 1408

# parted -s /dev/sdb unit MiB print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sdb: 2048MiB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number  Start    End      Size    Type      File system     Flags
 1      8.00MiB  808MiB   800MiB  primary   ext2            type=83
 2      808MiB   1208MiB  400MiB  primary   linux-swap(v1)  type=82
 3      1208MiB  1708MiB  500MiB  extended                  lba, type=0f
 5      1209MiB  1408MiB  199MiB  logical                   type=83

# parted -s /dev/sdb unit B print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sdb: 2147483648B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number  Start        End          Size        Type      File system     Flags
 1      8388608B     847249407B   838860800B  primary   ext2            type=83
 2      847249408B   1266679807B  419430400B  primary   linux-swap(v1)  type=82
 3      1266679808B  1790967807B  524288000B  extended                  lba, type=0f
 5      1267728384B  1476395007B  208666624B  logical                   type=83

# echo '1267728384 / 1024 / 1024' | bc -l
1209.00000000000000000000

Lesson learned:

Do not rely on parted's automated alignment
(in particular do not rely on parted's optimal alignment)
but calculate the right values manually and
use the right values in parted calls.

Contributor

jsmeix commented Feb 21, 2018

Only FYI:

Right now it happened to me that I got such a parted
"Warning: The resulting partition is not properly aligned for best performance.".

In my case it happened this way:

# parted -s /dev/sdb unit MiB print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sdb: 2048MiB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number  Start    End      Size    Type      File system     Flags
 1      8.00MiB  808MiB   800MiB  primary   ext2            type=83
 2      808MiB   1208MiB  400MiB  primary   linux-swap(v1)  type=82
 3      1208MiB  1708MiB  500MiB  extended                  lba, type=0f

# parted -s -a optimal /dev/sdb unit MiB mkpart logical 1208 1408
Warning: The resulting partition is not properly aligned for best performance.

# parted -s /dev/sdb unit MiB print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sdb: 2048MiB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number  Start    End      Size    Type      File system     Flags
 1      8.00MiB  808MiB   800MiB  primary   ext2            type=83
 2      808MiB   1208MiB  400MiB  primary   linux-swap(v1)  type=82
 3      1208MiB  1708MiB  500MiB  extended                  lba, type=0f
 5      1208MiB  1408MiB  200MiB  logical                   type=83

# parted -s /dev/sdb unit B print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sdb: 2147483648B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number  Start        End          Size        Type      File system     Flags
 1      8388608B     847249407B   838860800B  primary   ext2            type=83
 2      847249408B   1266679807B  419430400B  primary   linux-swap(v1)  type=82
 3      1266679808B  1790967807B  524288000B  extended                  lba, type=0f
 5      1266680320B  1476395519B  209715200B  logical                   type=83

# echo '1208 * 1024 * 1024' | bc -l
1266679808

# echo '1266680320 / 1024 / 1024' | bc -l
1208.00048828125000000000

# echo '1266680320 - 1266679808' | bc -l
512

I had specified the logical partition start with 1 MiB unit
at the same point where the extended partition starts
but parted knows that the first sector (i.e. the first 512 bytes)
of the extended partition cannot be used by the logical partition
so that parted automatically moved the actual logical partition start
by only 512 bytes which results an actual logical partition start point
that is "not properly aligned for best performance"
(whatever that exactly means in parted's opinion - see next paragraph)
and - as far as I know - there should be an at least 63 sectors gap
(i.e. at least 63 * 512 bytes = 32256 bytes) gap between the
extended partition start and the first logical partition start.

I think when parted can make such a "Warning" message
cf. http://blog.schlomo.schapiro.org/2015/04/warning-is-waste-of-my-time.html
parted must have some built-in "knowledge" what would be
"properly aligned for best performance" and then I wonder why
parted does not automatically align it "properly for best performance"
regardless that I had explicitly called parted with -a optimal
i.e. what the heck is parted's alignment type optimal meant for?

When I manually specify the logical partition start with 1 MiB unit
at one MiB more from where the extended partition starts it works:

# parted -s /dev/sdb rm 5

# parted -s /dev/sdb unit MiB mkpart logical 1209 1408

# parted -s /dev/sdb unit MiB print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sdb: 2048MiB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number  Start    End      Size    Type      File system     Flags
 1      8.00MiB  808MiB   800MiB  primary   ext2            type=83
 2      808MiB   1208MiB  400MiB  primary   linux-swap(v1)  type=82
 3      1208MiB  1708MiB  500MiB  extended                  lba, type=0f
 5      1209MiB  1408MiB  199MiB  logical                   type=83

# parted -s /dev/sdb unit B print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sdb: 2147483648B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number  Start        End          Size        Type      File system     Flags
 1      8388608B     847249407B   838860800B  primary   ext2            type=83
 2      847249408B   1266679807B  419430400B  primary   linux-swap(v1)  type=82
 3      1266679808B  1790967807B  524288000B  extended                  lba, type=0f
 5      1267728384B  1476395007B  208666624B  logical                   type=83

# echo '1267728384 / 1024 / 1024' | bc -l
1209.00000000000000000000

Lesson learned:

Do not rely on parted's automated alignment
(in particular do not rely on parted's optimal alignment)
but calculate the right values manually and
use the right values in parted calls.

@jsmeix

This comment has been minimized.

Show comment
Hide comment
@jsmeix

jsmeix Feb 23, 2018

Contributor

I think
#1733 (comment)
and
#1733 (comment)
prove that "rear recover" at least in migration mode
can result a changed and badly aligned partitioning.

Contributor

jsmeix commented Feb 23, 2018

I think
#1733 (comment)
and
#1733 (comment)
prove that "rear recover" at least in migration mode
can result a changed and badly aligned partitioning.

jsmeix added a commit that referenced this issue Mar 1, 2018

Merge pull request #1733 from jsmeix/replace_400_autoresize_disks_by_…
…420_autoresize_last_partitions_and_430_autoresize_all_partitions_issue1731

Major rework and changed default behaviour how ReaR behaves
in migration mode when partitions can or must be resized
to fit on replacement disks with different size,
see #1731 and the
related issue #102
The new default behaviour is that only the partition end value
of the last partition on a disk (and therefore its partition size)
may get changed if really needed but no partition start value
gets changed to avoid changes of the partition alignment.
The new 420_autoresize_last_partitions script implements
the new behaviour and the old 400_autoresize_disks was
renamed into 430_autoresize_all_partitions to still provide
the old behaviour if that is explicitly requested by the user
but the old behaviour may result unexpected changes
of arbitrary partitions on a disk.
The new config variables AUTORESIZE_PARTITIONS
AUTORESIZE_EXCLUDE_PARTITIONS
AUTOSHRINK_DISK_SIZE_LIMIT_PERCENTAGE
AUTOINCREASE_DISK_SIZE_THRESHOLD_PERCENTAGE
determine how ReaR behaves in migration mode
when partitions can or must be resized.
With AUTORESIZE_PARTITIONS='yes' the old behaviour is done.
With AUTORESIZE_PARTITIONS='no' no partition is resized by ReaR.
With the default AUTORESIZE_PARTITIONS='' at most the last active
partition on each active disk gets resized but only if really needed
which also depends on the settings of the other config variables above.
For details see default.conf and the two ' autoresize' scripts.
For some examples see #1733

@jsmeix jsmeix modified the milestones: ReaR future, ReaR v2.4 Mar 1, 2018

@jsmeix

This comment has been minimized.

Show comment
Hide comment
@jsmeix

jsmeix Mar 1, 2018

Contributor

With #1733 merged
this issue should be (hopefully) fixed, cf.
#1733 (comment)

But this means there is now a changed default behaviour
how ReaR behaves in migration mode when partitions can or must be
resized to fit on replacement disks with different size, cf. the merge commit comment
6414936

Contributor

jsmeix commented Mar 1, 2018

With #1733 merged
this issue should be (hopefully) fixed, cf.
#1733 (comment)

But this means there is now a changed default behaviour
how ReaR behaves in migration mode when partitions can or must be
resized to fit on replacement disks with different size, cf. the merge commit comment
6414936

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment