Adjust qcow2 image files to where 10GB really equals 10GB. #595

Closed
claflico opened this Issue Feb 20, 2015 · 3 comments

Projects

None yet

2 participants

@claflico

Don't know if it is supported or not but I download the qcow2 images for memcached & redis and import them into Openstack. They show to be <100MB on the dashboard however attempting to launch an instance on a 10GB flavor fails to due to the hard-drive being too small.

View the details of the qcow file to see:
qemu-img info osv-redis-memonly-v0.17.qemu.qcow2
image: osv-redis-memonly-v0.17.qemu.qcow2
file format: qcow2
virtual size: 10G (10842275840 bytes)
disk size: 30M
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false

10842275840 bytes actually equals 10.8423GB which if I adjust my flavors to be 11GB I am able to launch the instance but then I don't have consistency with the rest of my flavors.

I also don't see why if the qcow2 files is only 30M that why the virtual size needs to be 10GB.

Thanks.

@nyh
Contributor
nyh commented Feb 24, 2015

On Sat, Feb 21, 2015 at 1:48 AM, Cory Claflin notifications@github.com
wrote:

Don't know if it is supported or not but I download the qcow2 images for
memcached & redis and import them into Openstack. They show to be <100MB on
the dashboard however attempting to launch an instance on a 10GB flavor
fails to due to the hard-drive being too small.

View the details of the qcow file to see:
qemu-img info osv-redis-memonly-v0.17.qemu.qcow2
image: osv-redis-memonly-v0.17.qemu.qcow2
file format: qcow2
virtual size: 10G (10842275840 bytes)
disk size: 30M
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false

10842275840 bytes actually equals 10.8423GB which if I adjust my flavors
to be 11GB I am able to launch the instance but then I don't have
consistency with the rest of my flavors.

I also don't see why if the qcow2 files is only 30M that why the virtual
size needs to be 10GB.

This 10GB or 11GB is a completely arbitrary size that can be set in build.mk's
fs_size_mb (you can also set it in the "make" command line).
If you need your virtual disk to contain empty space for data - beyond the
30 MB taken by the application, then obviously you need the virtual disk to
be able to grow to more than 30 MB - and 10 GB was arbitrarily chosen.

But why is our 10GB bigger than 10GB?

First of all, our code in build.mk work in powers of 2, not powers of 10,
so our default size isn't 10,000,000,000 bytes, it is 10_1024_1024*1024
bytes.

This almost explains the size you're seeing, except for about 100MB. That
could further be explained, I think, by the "+" in the following command in
build.mk:

    $(call quiet, qemu-img resize $@ +$(fs_size_mb)M > /dev/null 2>&1)

I don't know why this "+" appears here (and I already noted that in a
comment in my proposed rewrite of the makefile).

I believe that to get exactly 10,000,000,000 bytes, you can modify the
above line to drop the "M" (so exact bytes will be specified) and the "+".

Of course, without any changes you can also use fs_size_mb=9000, and it
will be smaller than 10 GB :-)

Nadav Har'El
nyh@cloudius-systems.com

@nyh
Contributor
nyh commented Feb 24, 2015

Fixed on my makefile rewrite branch (see issue #571), removing the superfluous "+" as described above.

On that branch, we now get:

$ scripts/build image=rogue fs_size_mb=1024
$ qemu-img info build/release.x64/usr.img
...
virtual size: 1.0G (1073741824 bytes)

(note how 1024_1024_1024=1073741824, exactly).

Also added an option to set the size in bytes exactly. However, this number is rounded down to the nearest multiple of 512:

$ scripts/build image=rogue fs_size=100000000
$ qemu-img info build/release.x64/usr.img
...
virtual size: 95M (99999744 bytes)

(Note 99999744 is 100000000 rounded down to the nearest multiple of 512)

@claflico

Thanks.....I think...lol If what you did makes the file size smaller then I'll give it another try when the next qcow2 file gets released.

Not meaning to sound lazy and realize that this probably isn't the intended use of the project but I'm not wanting to have to build/compile anything. I would like to be able to just download the qcow2 files for memcache & redis, import them into OpenStack, and have ready to go instances that work with my existing 10GB flavors. If not doable, then thanks for replying and I'll keep building them by hand.

@avikivity avikivity added a commit that referenced this issue Mar 29, 2015
@nyh @avikivity nyh + avikivity build: Makefile rewrite
in v3: support setting CROSS_PREFIX in the environment as an an alternative
to setting arch or ARCH (which is still supported as well). For example,
someone can set (in environment or command line) CROSS_PREFIX=/some/prefix-
and this prefix will be used for the compilers - and also "arch" doesn't
need to be specified (it is figure out automatically by the given compiler).

This patch is a complete rewrite of OSv's build system. The rewrite had
two main goals:

1. Simplify the Makefile:

   Our old build systems had several makefiles including each other,
   playing tricks with the current directory and VPATH, dynamically rewriting
   makefiles, and running submakes. In the new build system, there is just
   one "Makefile" for building the entire OSv kernel. It's still not short,
   and could be further simplified in the future, but everything is in one
   file, and also better commented.

2. Separate kernel building from application building:

   In the old build system, we used "make" to do everything from building
   the OSv kernel, building various applications, and building images
   containing OSv and a collection of applications. This complicated the
   makefile, and resulted in unexpected build requirements (e.g., building
   OSv always built some Java tests and thus required Maven and a working
   Internet connection).

   In the new system, "make" only builds the OSv kernel, and "scripts/build"
   build applications and images (theoretically, Capstan could be used
   instead of scripts/build).

   Most "make" command lines that worked in the previous build system will
   continue to work unchanged if just changed to use "scripts/build".
   For example:

      scripts/build         # build image with default OSv application
      scripts/build image=rogue
      scripts/build modules=rogue
      scripts/build clean   # clean kernel and all modules
      scripts/build mode=debug # make parameters can also be given to build
      scripts/build check   # build image with tests, and run them

Additional benefits of this rewrite include:

1. Faster rebuilds. For example "touch loader.cc; scripts/build image=rogue"
   takes just 6 seconds (14 seconds previously). make after "make clean"
   (with ccache) is just 10 seconds (30 seconds previously).

2. It should be fairly easy to add, like we no have scripts/build, additional
   build scripts which will build different types of images using the same
   OSv kernel. One popularly requested option is to have the ability to
   create a bootfs-only image, without ZFS.

3. Some smaller improvements, like more accurate setting of the desired
   image size (#595), and supporting setting CROSS_PREFIX without additionally
   needing to specify ARCH.

Fixes #571
Fixes #595

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
[avi: adjust for new tests/tst-select-timeout.cc]
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
f37cef7
@avikivity avikivity added a commit that closed this issue Mar 29, 2015
@nyh @avikivity nyh + avikivity build: Makefile rewrite
in v3: support setting CROSS_PREFIX in the environment as an an alternative
to setting arch or ARCH (which is still supported as well). For example,
someone can set (in environment or command line) CROSS_PREFIX=/some/prefix-
and this prefix will be used for the compilers - and also "arch" doesn't
need to be specified (it is figure out automatically by the given compiler).

This patch is a complete rewrite of OSv's build system. The rewrite had
two main goals:

1. Simplify the Makefile:

   Our old build systems had several makefiles including each other,
   playing tricks with the current directory and VPATH, dynamically rewriting
   makefiles, and running submakes. In the new build system, there is just
   one "Makefile" for building the entire OSv kernel. It's still not short,
   and could be further simplified in the future, but everything is in one
   file, and also better commented.

2. Separate kernel building from application building:

   In the old build system, we used "make" to do everything from building
   the OSv kernel, building various applications, and building images
   containing OSv and a collection of applications. This complicated the
   makefile, and resulted in unexpected build requirements (e.g., building
   OSv always built some Java tests and thus required Maven and a working
   Internet connection).

   In the new system, "make" only builds the OSv kernel, and "scripts/build"
   build applications and images (theoretically, Capstan could be used
   instead of scripts/build).

   Most "make" command lines that worked in the previous build system will
   continue to work unchanged if just changed to use "scripts/build".
   For example:

      scripts/build         # build image with default OSv application
      scripts/build image=rogue
      scripts/build modules=rogue
      scripts/build clean   # clean kernel and all modules
      scripts/build mode=debug # make parameters can also be given to build
      scripts/build check   # build image with tests, and run them

Additional benefits of this rewrite include:

1. Faster rebuilds. For example "touch loader.cc; scripts/build image=rogue"
   takes just 6 seconds (14 seconds previously). make after "make clean"
   (with ccache) is just 10 seconds (30 seconds previously).

2. It should be fairly easy to add, like we no have scripts/build, additional
   build scripts which will build different types of images using the same
   OSv kernel. One popularly requested option is to have the ability to
   create a bootfs-only image, without ZFS.

3. Some smaller improvements, like more accurate setting of the desired
   image size (#595), and supporting setting CROSS_PREFIX without additionally
   needing to specify ARCH.

Fixes #571
Fixes #595

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
[avi: adjust for new tests/tst-select-timeout.cc]
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
f37cef7
@avikivity avikivity closed this in f37cef7 Mar 29, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment