Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Branch: master
Fetching contributors…

Cannot retrieve contributors at this time

executable file 661 lines (509 sloc) 26.694 kB
#!/usr/bin/env python2
#
LICENSE="""
Copyright (c) 2012 John Lane
MIT License
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
http://www.opensource.org/licenses/mit-license.php
"""
DESCRIPTION="""
rpi_mkimg : Make a Raspberry Pi SD Card Image
Describes and, optionally, makes a disk image with partitions
and filesystems prepared in accordance with the write and erase
geometries of the target device. Optionally copies files into
the new image from another image, directory or archive (tar file).
Quick Start
-----------
Those that don't want to read the detail can try this:
$ sudo rpi_mkimg --device=sdcard --copy=image --create
where 'sdcard' is the name of the sdcard device (e.g. mmcblk0)
and 'image' is the name of an image to copy (e.g.
archlinuxarm-13-06-2012.img)
Detail
------
A device and/or an image name can be given as command-line
arguments.
If a device is given (such as an SD card that will be used
to boot a Raspberry Pi), it will be used to set the attributes
that are used to determine the block-level layout of the image.
If no device is given then default values are used:
volume size = 2GiB
preferred erase size = 4MiB
These attributes are used to define a partitioning scheme
that ensures block alignment of the partitions and the file-
systems on those partitions. Two partitions are created:
boot : a FAT32 partition (at least 40Mb)
root : an ext4 partition
The partitioning scheme is displayed on standard output.
A command-line argument can be given which causes a disk image
to be created using the partitioning scheme. If this argument is
not given then this will not happen. This is to protect against
accidental writes.
The image can be written directly onto the device or it can
be written to a file that can be subsequently transferred onto
a suitable device using 'dd'.
If an image is specified then a new file will be created (replacing
any existing file of the same name) and the partitioning scheme
applied to it.
If an image is not specified but a device is specified then
the partitioning scheme will be applied to the device.
NOTE: All contents of the device or any existing image file
will be overwritten!
If neither image nor device are specified, the create argument
is ignored and nothing will be created.
Write permission to the image or device is required. Normally
the latter will require root permissions.
A command-line argument can be given which will copy files into
the created image. This mounts the new image's root filesystem
and then mounts its boot filesystem on top at /boot. The copy is
then performed into the mount point.
The files to be copied can be in another image file, directory
or archive (tar) file.
Block Alignment
---------------
The attribute used to define the device's block alignment is
called preferred erase size. The actual value can be overriden
using a command line argument.
Block alignment is performed on the partitions as well as on the
filesystems within them.
Partitions and Filesystems
--------------------------
Two partitions are created in accordance with the requirements
of the Raspberry Pi: A FAT32 partition containing a FAT16 boot
filesystem and a linux partition containing an ext4 root file-
system. Both partitions and filesystems are created in such a
way that their data blocks are aligned with those of the device.
By default:
- the boot partition size will be 40Mb but it may be larger than
this if the device's block sizes result in larger partitions;
- the root partition size will consume all of the
remaining space in the image;
- the total image size will be the size of any specified device
or 2GiB.
Default sizes may be overridden using command-line arguments:
- specifying a minimum size for a partition will ensure that the
partition is at least that size. If this this cannot be achieved
then the programme will exit with an appropriate message.
- specifying a maximum size for a partition will make the
partition no bigger than that maximum (subject to block alignment,
see below) and any remaining space in the image will be unused.
Partitions may be sized larger than a specified maximum if this is
necessary to achieve block alignment.
Operational requirements
------------------------
Arch Linux package requirements:
parted : to make partitions
dosfstools : to make vfat partition
python-pexpect : Python Expect used to run commands
The loop device is used to mount image files. Ensure the "loop" kernel
module is loaded: "modprobe loop" should achieve this.
Commands requiring privileged entitlements are executed with sudo.
(c) John Lane 2012-08-03.
Licensed under the MIT License.
Part of the rpi-utilities:
https://github.com/johnlane/rpi-utils
"""
# References:
#
import sys
import os
import pexpect
import argparse
import atexit
import tempfile
import pwd
def align(sector,grain):
if args.align == True:
excess = sector%grain
sector = sector if excess == 0 else sector + grain - excess
return sector
def print_partition_info(name,start_sector,end_sector):
size_mib = (end_sector - start_sector + 1) * 512 / (1<<20)
print "------------------------------------------------------------"
print " Partition '%s':" % name
print " Start sector : %d" % start_sector
print " End sector : %d" % end_sector
print " 512 byte sectors : %d" % (end_sector - start_sector + 1)
print " 1KiB Blocks : %d" % ((end_sector - start_sector + 1) / 2)
print " Size : %d MiB" % ((end_sector - start_sector + 1) * 512 / (1<<20))
print "------------------------------------------------------------"
def gib(n):
return (float(n)/(1<<30))
def mib(n):
return (float(n)/(1<<20))
def kib(n):
return (float(n)/(1<<10))
def print_bytes(label,value):
llen=len(label)
print "%s : %d bytes" % (label,value)
if value>=(1<<10) and value<(1<<20): print "%s : %g Kib" % (''.rjust(llen),kib(value))
if value>=(1<<20): print "%s : %g Mib" % (''.rjust(llen),mib(value))
if value>=(1<<30): print "%s : %g Gib" % (''.rjust(llen),gib(value))
def run_command(command):
output, exit_status = pexpect.run(command, withexitstatus=1)
output = output.strip()
return output, exit_status
def do_or_die(command,error_message="Command Failed"):
output, exit_status = run_command(command)
if exit_status != 0:
abort("%s : %s" % (error_message, output))
return output
def writable_or_die(d):
if os.access(d,os.W_OK) != True:
abort("No write access to %s" % d)
def create_loop_device(d):
loop_device = do_or_die("losetup -f","Unable to get a loop device")
debug("Setting up loop device %s for %s" % (loop_device,d))
do_or_die("losetup -P %s %s" % (loop_device,d), "Unable to create loop device")
return loop_device
def destroy_loop_device(d):
do_or_die("losetup -d %s" % d,"Unable to destroy loop device")
def abort(error_message):
sys.exit("%s. Cannot continue" % error_message)
##############################################################################################
# A very primitive message logger
LOG_DEBUG = 'debug'
LOG_VERBOSE = 'verbose'
def verbose(message): log(LOG_VERBOSE,message)
def debug(message): log(LOG_DEBUG,message)
def log(level,message):
if args.loglevels != None and level in args.loglevels: print "(%s) %s" % (level,message)
def debug_enabled(): return args.loglevels != None and LOG_DEBUG in args.loglevels
def verbose_enabled(): return args.loglevels != None and LOG_VERBOSE in args.loglevels
##############################################################################################
def allocate_sectors(first_sector,available_sectors,sector_alignment_grain,minimum_kb,maximum_kb,align_start=True):
min_sectors = minimum_kb / 512
max_sectors = maximum_kb / 512
if minimum_kb < ( min_sectors * 512 ): min_sectors += 1
if maximum_kb < ( max_sectors * 512 ): max_sectors += 1
if max_sectors == 0: max_sectors = available_sectors
debug("requested minimum %s and maximum %s sectors" % (min_sectors,max_sectors))
# align the first sector except unless requested not to
start_sector = align(first_sector,sector_alignment_grain) if align_start else first_sector
debug("allocating sectors, %d available starting at sector %d" % (available_sectors,start_sector))
debug("starting sector %d aligned to %d" % (first_sector,start_sector))
available_sectors -= (start_sector - first_sector)
debug("%d available sectors after front alignment" % available_sectors)
if available_sectors < min_sectors:
abort("Unable to allocate %d sectors (%d available)" % (min_sectors,available_sectors))
num_sectors = max_sectors if available_sectors >= max_sectors else available_sectors
debug( "allocated %d sectors" % num_sectors)
next_sector = start_sector + num_sectors
debug("the next free sector is %d" % next_sector)
# Unless we've used up all sectors, align the next free sector and extend allocation
if next_sector == start_sector + available_sectors:
# There are no more free sectors; the end sector is one less than the total number of sectors
# fdisk numbers LBA sectors from 0 but sector 0 is 'hidden' and contains the MBR. Sector
# numbers for partitions start at 1. Because there is a sector 0 the number of the last
# sector is one less than the total number of sectors. The value 'next sector' will be one
# greater than the total number of sectors, so we subtract 2 to get the last sector.
#
end_sector = next_sector - 1
next_sector = 0
available_sectors = 0
elif next_sector > start_sector + available_sectors:
# This case should never arise
abort("Unexpected sector number %d, which is greater than the total number of sectors, %d" % (next_sector,start_sector + available_sectors))
else:
# Align the next free sector; the end sector is the one before that
next_sector = align(next_sector,sector_alignment_grain)
end_sector = next_sector - 1
debug("start %d" % start_sector)
debug("end %d" % end_sector)
debug("num %d" % num_sectors)
debug("next %d" % next_sector)
return start_sector, end_sector, next_sector, available_sectors - (end_sector +1 - start_sector)
def main(name, argv):
def cleanup():
try:
if loop_device: destroy_loop_device(loop_device)
if copy_loop_device: destroy_loop_device(copy_loop_device)
except NameError as e:
pass # move along, nothing to do
atexit.register(cleanup)
# Default sizes
pes = 4<<20 # 4MiB
#wbs = 8<<10 # 8KiB (not used)
vs = 2<<30 # 2GiB
# Parition sizing minima/maxima
bp_size_minimum = 80<<20 # 80MiB
bp_size_maximum = 80<<20 # 80MiB
rp_size_minimum = 1<<30 # 1GiB
rp_size_maximum = 0 # No limit
# Arguments
global args
parser = argparse.ArgumentParser(description=DESCRIPTION, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument('--license', action='store_true', default=False, dest='license',
help='show the MIT License')
parser.add_argument('-d', '--device', action='store', dest='device',
help='Specify a device name (e.g. mmcblk0)')
parser.add_argument('-i', '--image', action='store', dest='image',
help='Specify an image file name(e.g. myimage.img)')
parser.add_argument('--copy', action='store', dest='copy',
help='Copy into the image from another image, directory or archive (tar) file ')
parser.add_argument('-c', '--create', action='store_true', default=False, dest='create',
help='Create (overwrites existing data)')
parser.add_argument('-v', '--verbose', action='append_const', const=LOG_VERBOSE, dest='loglevels',
help='Enable verbose message output')
parser.add_argument('--debug', action='append_const', const=LOG_DEBUG, dest='loglevels',
help='Enable debug message output')
parser.add_argument('--boot-minimum', action='store', dest='bp_size_minimum',
help='Override the boot partition minimum size default (%d)'%bp_size_minimum)
parser.add_argument('--boot-maximum', action='store', dest='bp_size_maximum',
help='Override the boot partition maximum size default (%d)'%bp_size_maximum)
parser.add_argument('--root-minimum', action='store', dest='rp_size_minimum',
help='Override the root partition minimum size default (%d)'%rp_size_minimum)
parser.add_argument('--root-maximum', action='store', dest='rp_size_maximum',
help='Override the root partition maximum size default (%d)'%rp_size_maximum)
parser.add_argument('--first-sector', action='store', dest='first_sector',
help='Specify a first sector to use (default is sector 1)')
parser.add_argument('--size', action='store', dest='volume_size',
help='Specify the size of the whole disk in bytes (default is %d bytes)'%vs)
parser.add_argument('--erase-block-size', action='store', dest='erase_block_size',
help='Specify the erase block size bytes (default is %d bytes or that of the device specified with -d)'%vs)
#parser.add_argument('--write-block-size', action='store', dest='write_block_size',
# help='Specify the write block size bytes (default is %d bytes or that of the device specified with -d)'%vs)
parser.add_argument('--no-align', action='store_false', default=True, dest='align',
help='Disable write block alignment')
parser.add_argument('--no-align-first', action='store_false', default=True, dest='align_first_partition',
help='Do not align the first partition (it will start at first-sector)')
args = parser.parse_args()
if args.license == True:
print LICENSE
return 0
if args.bp_size_minimum != None: bp_size_minimum = int(args.bp_size_minimum)
if args.bp_size_maximum != None: bp_size_maximum = int(args.bp_size_maximum)
if args.rp_size_minimum != None: rp_size_minimum = int(args.rp_size_minimum)
if args.rp_size_maximum != None: rp_size_maximum = int(args.rp_size_maximum)
# Preferred Erase Size
if args.erase_block_size != None:
pes = int(args.erase_block_size)
elif (args.device != None):
try:
device = args.device
device_sys_path = "/sys/class/block/%s" % (device, )
pes = int(file(device_sys_path+"/device/preferred_erase_size").read())
print "Using device %s" % device
vss = int(file(device_sys_path+"/size").read())
vs = vss * 512
except IOError as e:
print "Device %s not found: Using default values." % device
else:
print "No Device specified: Using default values."
# Write Block Size
#if args.write_block_size != None: wbs = int(args.write_block_size)
# Volume size in sectors
if args.volume_size != None: vs = int(args.volume_size)
vss = vs / 512 # vs in sectors
# First available sector (sector 0 is reserved for the MBR which contains the partition table)
first_sector = 1 if args.first_sector == None else int(args.first_sector)
print "------------------------------------------------------------"
print "Volume Size : %d sectors" % vss
print_bytes("Volume Size",vs)
if bp_size_minimum>0: print_bytes("Boot Partition Minimum Size",bp_size_minimum)
if bp_size_maximum>0: print_bytes("Boot Partition Maximum Size",bp_size_maximum)
if rp_size_minimum>0: print_bytes("Root Partition Minimum Size",rp_size_minimum)
if rp_size_maximum>0: print_bytes("Root Partition Maximum Size",rp_size_maximum)
#print_bytes("Write Block Size",wbs)
print_bytes("Preferred Erase Size",pes)
# sector alignment grain in sectors (512 bytes per sector)
sector_alignment_grain = pes/512
print "Sector Alignment Grain : %d / 512 = %d sectors" % (pes,sector_alignment_grain)
# Total number of available sectors includes the first sector (which is sector 0)
# the unused sectors are 0 thru (first_sector -1). Sector 0 is reserved for the MBR.
available_sectors = vss - first_sector
print "Starting at sector %d of %d total sectors" % (first_sector,vss)
print "The %d available sectors are %d thru %d" % (available_sectors,first_sector,vss-1)
print "%d sectors will be skipped (sectors 0 thru %d)" % (first_sector,first_sector-1)
print
print " Partition description"
# Boot partition
bp_start_sector, bp_end_sector, next_sector, available_sectors = allocate_sectors(first_sector,available_sectors,sector_alignment_grain,bp_size_minimum,bp_size_maximum,args.align_first_partition)
print_partition_info("boot",bp_start_sector, bp_end_sector)
# Root partition
rp_start_sector, rp_end_sector, next_sector, available_sectors = allocate_sectors(next_sector,available_sectors,sector_alignment_grain,rp_size_minimum,rp_size_maximum)
print_partition_info("root",rp_start_sector,rp_end_sector)
# Report any unused sectors
if available_sectors > 0:
print "Partitioning scheme leaves %s sectors unused" % available_sectors
# Only proceed if create option given
if args.create == False:
return 0
# must be root to go beyond here
if os.geteuid() != 0:
abort('Partitioning requires root privileges (and you are not root)')
# Where to write data
if args.image != None:
dest = args.image
elif args.device != None:
dest = args.device
dest_device = "/dev/%s" % dest
else:
print "Nothing to Create: neither device nor image specified."
return 0
# If dest is an image, create the image file and set up a loop device for it
if dest == args.image:
try:
if os.path.exists(dest): os.remove(dest) # remove any old image first
if pexpect.run('/bin/bash -c "df -T . | tail -1 | awk \'{print $2}\'"').rstrip() == "ext4":
debug("Using fallocate to create %s (%d bytes)" % (dest,vs))
pexpect.run("fallocate -l %d %s" % (vs,dest))
else:
debug("Using truncate to create %s (%d bytes)" % (dest,vs))
pexpect.run("truncate -s %d %s" % (vs,dest))
# make file owned by actual user (e.g. not root if invoked with sudo)
user = pexpect.run('logname').strip()
if user.find('no login name') == -1 and user != os.environ['USER']:
uid = pwd.getpwnam(user).pw_uid
gid = pwd.getpwnam(user).pw_gid
os.chown(dest,uid,gid)
debug("Changed %s ownweship to user %s (%d:%d)" % (dest,user,uid,gid))
loop_device = create_loop_device(dest)
dest_device = loop_device
dest_size = os.path.getsize(dest)
if dest_size != vs:
os.remove(dest)
abort("Created %s has incorrect size %d (expected %d)"%(dest,dest_size,vs))
except IOError as e:
abort("Unable to create %s" % dest)
# Make sure we can write to the destination
writable_or_die(dest_device)
debug("Will write to %s as %s" % (dest,dest_device))
# Get user confirmation
print "About to write new partition table to %s. Any existing data will be lost!!!!" % dest
if (raw_input("Enter 'yes' to continue (anything else quits!)...") != 'yes'):
return 0
# Do partitioning with parted
print "parted -s %s mktable msdos" % dest_device
do_or_die("parted -s %s mktable msdos" % dest_device, "Failed to make new partition table")
do_or_die("parted -s %s unit s mkpart primary fat32 %d %d" % (dest_device,bp_start_sector,bp_end_sector), "Failed to create boot partition")
do_or_die("parted -s %s unit s mkpart primary ext2 %d %d" % (dest_device,rp_start_sector,rp_end_sector), "Failed to create root partition")
# Print the partition table
if verbose_enabled(): do_or_die("parted %s unit s print" % dest_device)
# Recreate any loop device so partition devices are available
try:
if loop_device:
destroy_loop_device(loop_device)
loop_device = create_loop_device(dest)
except NameError as e:
pass # move along, nothing to do
# Partition filesystem devices
dest_p1 = dest_device+"p1"
dest_p2 = dest_device+"p2"
# boot filesystem to be FAT16, aligned
#
# vfat args:
# -I Don't complain about using whole device
# -F 16 FAT size is 16 bit
# -n boot volume label is 'boot'
# -s 16 sectors per cluster (16 * 512 = 8KiB)
# -v Verbose output
# -R number of sectors to reserve
print "Creating boot filesystem"
# Make sure we can write to the partition
writable_or_die(dest_p1)
# Make unaligned FAT filesystem to establish FAT size
mkfs = pexpect.spawn("mkfs.vfat -I -F 16 -n boot -s 16 -v %s" % dest_p1)
if debug_enabled(): mkfs.logfile = sys.stdout
mkfs.expect('FAT size is ([0-9]+) sectors')
fat_sectors = int(mkfs.match.groups()[0])
mkfs.expect('Volume ID is (.*), volume label (.*)')
vol_id, vol_name = mkfs.match.groups()
mkfs.expect(pexpect.EOF)
# make aligned FAT filesystem
if args.align == True and args.align_first_partition == True:
debug("There are %d sectors in the FAT" % fat_sectors)
# There are two FATS, work out total size in bytes
fat_bytes = fat_sectors * 512 *2
debug("FAT sectors total %d bytes" % fat_bytes)
# Work out how many sectors to reserve to achieve alignment
reserved_sectors = (pes - fat_bytes) / 512
debug("Reserve %d sectors to achieve alignment" % reserved_sectors)
# Make aligned FAT filesystem
mkfs = pexpect.spawn("mkfs.vfat -I -F 16 -n boot -s 16 -R %d -v %s" % (reserved_sectors, dest_p1))
if debug_enabled(): mkfs.logfile = sys.stdout
mkfs.expect('Volume ID is (.*), volume label (.*)')
vol_id, vol_name = mkfs.match.groups()
mkfs.expect(pexpect.EOF)
print "Created volume %s id %s" % (vol_id, vol_name.strip())
uuid = pexpect.run("blkid -o value -s UUID %s" % dest_p1).strip()
print "Volume UUID: %s" % uuid
# root partition to be ext4, aligned
#
# ext4 args:
# -O ^has_journal disable journalling
# -E stride=2 treat 2 blocks as 1
# stripe-width=512
# -b
# -L root
print "Creating root filesystem"
# Make sure we can write to the partition
writable_or_die(dest_p2)
ext4_block_size = 4096
ext4_stride = 2
ext4_stride_size = ext4_block_size * ext4_stride
ext4_stripe_width = pes / ext4_stride_size
mkfs = pexpect.spawn("mkfs.ext4 -O ^has_journal -E stride=%d,stripe-width=%d -b %d -L root %s" % (ext4_stride, ext4_stripe_width, ext4_block_size, dest_p2))
if debug_enabled(): mkfs.logfile = sys.stdout
mkfs.expect(pexpect.EOF)
uuid = pexpect.run("blkid -o value -s UUID %s" % dest_p2).strip()
print "Volume UUID: %s" % uuid
if args.copy != None:
copy = args.copy
# Mount the image
dest_mount_point = tempfile.mkdtemp()
debug("Mounting %s on %s" % (dest_p2,dest_mount_point))
pexpect.run("mount %s %s" % (dest_p2,dest_mount_point))
os.mkdir(dest_mount_point+"/boot")
debug("Mounting %s on %s" % (dest_p1,dest_mount_point+"/boot"))
pexpect.run("mount %s %s" % (dest_p1,dest_mount_point+"/boot"))
copy_type = pexpect.run("file %s" % copy).strip()
if "x86 boot sector" in copy_type:
print "Copying from image %s ..." % copy
copy_loop_device = create_loop_device(copy)
copy_boot_device = copy_loop_device+"p1"
copy_root_device = copy_loop_device+"p2"
copy_mount_point = tempfile.mkdtemp()
debug("Mounting %s on %s" % (copy_root_device,copy_mount_point))
debug("Mounting %s on %s" % (copy_boot_device,copy_mount_point+"/boot"))
pexpect.run("mount %s %s" % (copy_root_device,copy_mount_point))
pexpect.run("mount %s %s" % (copy_boot_device,copy_mount_point+"/boot"))
debug("Copying from %s to %s" % (copy_mount_point,dest_mount_point))
os.system("cd %s ; cp -a * %s" % (copy_mount_point,dest_mount_point))
debug("Unmounting %s" % copy_boot_device)
os.system("umount %s" % copy_boot_device)
debug("Unmounting %s" % copy_root_device)
os.system("umount %s" % copy_root_device)
debug("Removing directory %s" % copy_mount_point)
os.rmdir(copy_mount_point)
elif "directory" in copy_type:
print "Copyng from directory %s " % copy
os.system("cd %s ; cp -a * %s" % (os.path.abspath(copy),dest_mount_point))
elif "tar archive" in copy_type or "compressed data" in copy_type:
if "compressed data" in copy_type: print "File is compressed (assuming compressed tar file)"
print "Copying from tar file %s " % copy
os.system("cd %s ; tar xvf %s" % (dest_mount_point, os.path.abspath(copy)))
else:
print "Don't know how to copy %s" % copy
# Unmount the image
debug("Unmounting %s" % dest_p1)
pexpect.run("umount %s" % dest_p1) # /boot
debug("Unmounting %s" % dest_p2)
pexpect.run("umount %s" % dest_p2) # /
print "done"
sys.exit(main(sys.argv[0], sys.argv[1:]))
Jump to Line
Something went wrong with that request. Please try again.