Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Commits on Sep 4, 2013
  1. @kazum @liuy

    add helper function to count data objects in vdi

    kazum authored liuy committed
    This also removes redudant object checks in dog, and improves the
    performance of 'dog vdi list' when the vdi disk size is not so big.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
    Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Commits on Sep 2, 2013
  1. @kazum @liuy

    sheep: use sd_hash instead of fnv_64a_buf

    kazum authored liuy committed
    This patch uses sd_hash instead of fnv_64a_buf for better object
    dispersion.
    
    This also fixes the problem that we are using the number of disks for
    calculating virtual disk ids.  Without this patch, virtual disk ids of
    the existing disks will be changed for each we plug/unplug a disk; a
    lot of object movement will happen.
    
    Please note that this patch breaks backward compatibility because this
    changes a hash function of our consistent hashing algorithm, which
    means that object location will be also changed.  The upgrade code for
    the existing users will be appeared before releasing the next sheepdog
    release.
    
    I've not changed some hash calculation which is used for getting
    object file names (vdi object name and attribute object name).  It is
    because changing them makes it very hard to implement the upgrade code
    and forces us to modify qemu client codes too.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
    Signed-off-by: Liu Yuan <namei.unix@gmail.com>
  2. @kazum @liuy

    add sd_hash to generate more dispersed hash values

    kazum authored liuy committed
    I noticed that adding one more fnv_64a calculation like
    
      hval = fnv_64a_64(hval, hval)
    
    generates much more dispersed hash values.
    
    This patch introduces sd_hash(), which does the above calculation
    before returning the result.  This also add sd_hash_next(), which is
    intended to be used for virtual nodes and virtual disks.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
    Signed-off-by: Liu Yuan <namei.unix@gmail.com>
  3. @kazum @liuy

    optimize fnv hash

    kazum authored liuy committed
    FNV hash author says that a expression of shifts and adds is faster
    than FNV_prime multiply (*).  However, with the recent GCC and
    hardware, it looks like the FNV_prime multiply generates much faster
    code.
    
    This patch uses FNV_64_PRIME to generate codes, and adds fnv_64a_64 to
    create a hash value from 64 bit integers faster.  Note that this patch
    is just for optimization and will not change the generated hash
    values.
    
    (*) http://www.isthe.com/chongo/tech/comp/fnv/#gcc-O3
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
    Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Commits on Aug 12, 2013
  1. @kazum @liuy

    collie: remove master column from node list

    kazum authored liuy committed
    We removed a master node from cluster drivers, so the master column in
    the node list is meaningless now.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
    Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Commits on Jul 13, 2013
  1. @kazum @liuy

    clean up logger.h and util.h include

    kazum authored liuy committed
    This also fixes a compiler error which happens when NDEBUG is not
    defined.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
    Signed-off-by: Liu Yuan <namei.unix@gmail.com>
Commits on May 14, 2013
  1. @kazum @liuy

    sheep: add operation to get object sha1 digest

    kazum authored liuy committed
    Currently, we compare entire object contents to check whether there is
    no difference among them.  However, this wastes a lot of network
    bandwidth and some operations like vdi check or object recovery take
    too long time to complete.
    
    This adds an operation SD_OP_GET_HASH to get the sha1 digest of the
    object, and enables us to do faster vdi check and recovery.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
    Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Commits on May 2, 2013
  1. @kazum @liuy

    sheep: return SD_RES_READONLY when write requests are sent to snapshots

    kazum authored liuy committed
    This can notify the VM that it has to reload the vdi object.  In
    addition, this can also prevent snapshot objects from being updated
    wrongly.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
    Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
  2. @kazum @liuy

    sheep: store snapshot vid information in memory

    kazum authored liuy committed
    This renames vdi_copy to vdi_state and adds a new field 'snapshot' to
    it so that we can know whether the vdi is snapshot or not.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
    Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
  3. @kazum @liuy

    sheep: make oid_to_vid support any kind of objects

    kazum authored liuy committed
    get_obj_copy_number() doesn't take into account vmstate objects.  This
    fixes the problem by changing oid_to_vid() so that it supports any
    kind of objects.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
    Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
  4. @kazum @liuy

    move vdi_is_snapshot() to sheepdog_proto.h

    kazum authored liuy committed
    This allows us to use the macro outside of sheep/vdi.c.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
    Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
  5. @kazum @liuy

    remove sd_*_req and sd_*_rsp

    kazum authored liuy committed
    This patch removes all the sd_*_req and sd_*_rsp in internal_proto.h,
    and adds the necessary fields to sd_req and sd_rsp.  It is not good to
    include internal protocols in sheepdog_proto.h, but the change can
    decrease many casting, improve type safety, and simplify codes a lot.
    
    This also checks the size of sd_req and sd_rsp so that we don't break
    them.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
    Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Commits on Oct 19, 2012
  1. @kazum @liuy

    add const where possible

    kazum authored liuy committed
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
    Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Commits on Oct 6, 2012
  1. @kazum

    use bool for boolean variables

    kazum authored
    This improves code readability.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
  2. @kazum

    trim redundant zero bytes of network and disk I/O data

    kazum authored
    This will save a lot of network and disk I/Os especially when
    recoverying sparse objects.
    
    This updates the protocol version between sheep and other programs,
    but the older one is also supported.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
  3. @kazum

    make sd_rsp 64 bit aligned

    kazum authored
    Without this patch, the header size will end up being changed if we
    add a 64 bit field to sd_rsp->obj or sd_rsp->vdi.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Commits on Sep 21, 2012
  1. @kazum @liuy

    introduce SD_FLAG_CMD_DIRECT to bypass object cache

    kazum authored liuy committed
    Currently, collie commands also use object cache if it is enabled.
    This causes problems because sheep doesn't assume that other nodes
    also creates cache data of the same opened VDIs.  If we allow it, data
    inconsistency problems happen when another node updates the cached
    objects.
    
    This patch forces collie commands to bypass object cache to avoid the
    problem, but vdi read and write still use object cache because they
    are used to emulate VM I/Os.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
    Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Commits on Jul 9, 2012
  1. @kazum @liuy

    add a helper function to calculate the max size of objects

    kazum authored liuy committed
    This prepares for the next patch.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
    Signed-off-by: Liu Yuan <tailai.ly@taobao.com>
Commits on Nov 18, 2011
  1. @kazum

    reduce the maximum size of vdi attributes from 4 MB to 64 KB

    kazum authored
    This allows us to make simple_store_read()/write() fail when it cannot
    read/write full length data.
    
    This patch can also remove SD_FLAG_CMD_TRUNCATE.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Commits on Oct 20, 2011
  1. @kazum

    add vdi creation time to the vdi attribute header

    kazum authored
    Currently, Sheepdog doesn't clean up vdi attributes when we delete the
    vdi.  So if we create a new vdi whose name was used before, we could
    read the old attribute.
    
    This patch adds a vdi creation time to the attribute header and checks
    whether the attribute belongs to the current vdi.  Note that this is a
    transitional approach. In future, we should remove vdi attributes when
    we remove the vdi.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Commits on Aug 10, 2011
  1. @kazum

    collie: add vdi creation support

    kazum authored
    $ collie vdi create -h
    vdi create - create a image
    
    Usage:
      collie vdi create <vdiname> <size> [-P] [-a address] [-p port] [-h]
    
    Command parameters:
      -P, --prealloc          preallocate all the data objects
      -a, --address           specify the daemon address (default: localhost)
      -p, --port              specify the daemon port
      -h, --help              display this help and exit
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Commits on Apr 6, 2011
  1. @kazum

    suppor direct IO

    kazum authored
    O_SYNC on ext3/4 with barrier=1 and btrfs causes severe performance
    problems.  This introduces -D option to sheep command line arguments,
    and enables O_DIRECT for data objects.
    
    TODO:
      Enables O_DIRECT for other kinds of objects (e.g. vdi objects)
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
  2. @kazum

    distinguish different kinds of objects

    kazum authored
    Sheepdog has four kinds of objects:
    
      vdi object      - contains metadata of vdi
      data object     - contains actual data of vdi
      vmstate object  - contains vmstate data which is used for live snapshot
      vdi attr object - contains vdi attributes
    
    This patch distinguishes these objects and makes codes easy to maintain.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Commits on Mar 8, 2011
  1. @kazum

    update copyright year to 2011

    kazum authored
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Commits on Feb 15, 2011
  1. @kazum

    add vdi attributes support

    kazum authored
    It is useful to store metadata associated with virtual disks.  This
    patch adds support for vdi attributes.
    
    Usage:
     $ collie vdi getattr VDINAME KEY                     # get value
     $ collie vdi setattr VDINAME KEY [-x] VALUE          # set value
     $ collie vdi setattr VDINAME KEY [-x] < VALUEFILE    # set value from stdin
     $ collie vdi setattr VDINAME KEY -d                  # delete attribute
    
    -x is an exclusive option; if you set the option and the key already
    exists, the operation fails.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
  2. @kazum

    set the number of copies in the response of vdi operations

    kazum authored
    We need to know the number of replications to read/write objects
    directly.  This patch sets the number in the response of vdi
    operations, and avoid redundant accesses to the vdi objects.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Commits on Feb 1, 2011
  1. @kazum

    collie: avoid reading entire vdi object

    kazum authored
    This improves performance of some collie commands significantly.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Commits on Dec 25, 2010
  1. @kazum

    sheep: fix data consistency when reading objects for the first time

    kazum authored
    If total node failure happens, data consistency of replicated objects
    could be broken.  This patch overwrites replicated objects with the
    same data and recovers the data consistency when qemu reads the
    objects for the first time.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Commits on Jun 18, 2010
  1. @kazum

    sheep: add snapshot tag support

    kazum authored
    This patch enables us to add a snapshot name when creating a snapshot.
    
    This is mutually depends on the client patch I'll send later to the
    qemu-devel.  You can also get these from the latest git tree:
    
      server:
        git://sheepdog.git.sourceforge.net/gitroot/sheepdog/sheepdog next
      client:
        git://sheepdog.git.sourceforge.net/gitroot/sheepdog/qemu for-block
    
    usage:
      $ qemu-img snapshot -c tagname sheepdog:linux
    
      $ ./collie/collie vdi list
        name        id    size    used  shared    creation time   vdi id
      ------------------------------------------------------------------
      s linux        1  2.0 GB   48 MB  0.0 MB 2010-06-18 20:40   a5d05d
        linux        2  2.0 GB  0.0 MB   48 MB 2010-06-18 21:23   a5d05e
    
      $ qemu-img snapshot -l sheepdog:linux
      Snapshot list:
      ID        TAG                 VM SIZE                DATE       VM CLOCK
      1         tagname                   0 2010-06-18 21:23:32   00:00:00.000
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Commits on May 12, 2010
  1. @kazum

    move some request result codes from sheep.h to sheepdog_prog.h

    kazum authored
    Qemu can get these result codes.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Commits on May 10, 2010
  1. @kazum

    collie: return SD_NO_SPACE when there is no free space

    kazum authored
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Commits on May 9, 2010
  1. @kazum

    check protocol version

    kazum authored
    We should avoid wrong version members coming in the cluster.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
  2. @kazum

    remove unused request header fields

    kazum authored
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
  3. @kazum

    clean up header files

    kazum authored
    Header files about sheepdog protocols are grouped into three.
    
    include/sheepdog_proto.h - used between server and VM
    include/collie.h         - used in the server (including shepherd)
    collie/collie_priv.h     - internal to collie daemons
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Commits on May 7, 2010
  1. @kazum

    collie: reject invalid node joining to the sheepdog

    kazum authored
    When joining nodes have a wrong epoch information, sheepdog
    should reject the nodes and return the error result.
    
    Signed-off-by: MORITA Kazutaka <morita.kazutaka@lab.ntt.co.jp>
Something went wrong with that request. Please try again.