Container Format

About

This section gives you an idea about how MigrationContainer is technically implemented. It describes the key aspects of (de-)serialization of container headers and content allocation.

Single-Part Containers

The following figure depicts the basic structure of an unparted migration container, which is represented by a single physical file:

The Start- and ContentHeader contain a set of predefined fields, which serve to provide all necessary information to properly process and interpret the container’s content. All implementations of migration containers must strictly comply with these:

Header Name	Description
StartHeader	The StartHeader serves as entry point to the container. It provides all necessary information to initiate further processing.
ContentHeader	Provides all required information for retrieval and restoration of thecontainer’s contents. It holds the content’s meta data - This is whereall file attributes and directory hierarchies are stored.
ApplicationHeader	The ApplicationHeaders are optional. A container may hold an arbitrary number of these, however they always must be located in between the ContentHeader and the body. The ApplicationHeaders are meant to contain additional information which is not directly related to the content, but still required for the proper integration of data into the target system - This is where all account related information are stored.
Body	Contains all the binary data to be carried. It does not contain any meta data - This is where all files and versions are stored

The StartHeader's fields are defined as follows:

Field Name	Description
MagicNumbers	A fixed byte sequence, positioned at the very beginning of the container.
Md5Hash	Contains the the calculated MD5 hash of the entire container file excluding the MagicNumbers and of course the hash itself.
ContainerID	A byte sequence uniquely identifying a migration container.
PartNumber	The part number of the current container file.
Parts	The number of files a container consists of.
NextHeaderLength	The length of the subsequent header.
ContentLength	The length of the associated content inside the container’s body.
ContentOffset	The content’s distance to the body’s start position.
OriginalName	The original identifier that was used for the content on the source system.

The MagicNumbers are used for distinct and quick identification of a container file without being forced to analyse thee entire binary structure. The Md5Hash is used to detect corruptions to the container file. The ContainerID , PartNumber and Parts fields are required for parted container files to associate all related parts. This also makes the container immune to errors based to corrupted file names. The fields NextHeaderLength, ContentLength and ContentOffset are required for (de-)serialization and content allocation. For more information on these please refer to section 4.3 Serialization of my thesis.

Multi-Part Containers

Due to the limitations of file systems or network transport a migration container may need to be split into chunks. The following picture illustrates how this is accomplished:

The partition of migration containers must follow a few simple rules in order to preserve consistency of data:

1. All headers including the ApplicationHeaders must fit into a single file. This means, that the maximum allowed header size must not be smaller than the overall length of these headers.
2. Every subsequent file with a part number greater than 1 must also have a StartHeader to be uniquely identified.
3. Content that does not entirely fit into a container file must be continued in the subsequent container file.

Explanation

The file containing the Content- and ApplicationHeaders is called the main part, since it provides all required information to process the container. It always has the part number 0 and does not have any numbers in its file extension. In cases where an extremely low file size is chosen, the main part does not have a body, because it did not fit. Then the content is simply put into the next container file. All other parts except the main part are called related parts. In contrast to the main part their file extension always contains a part number. From the technical point of view this actually not required, since the number is also contained inside the StartHeader. However it makes non-programmatic management of parted containers easier and thus increases usability when the part number is exposed. The presence of the StartHeader in each file is required for several reasons. First of all it allows to identify a file as part of a certain container by its ContainerID. The part number allows to arrange and read all parts in the correct order and the MD5 hash allows to determine corruption of data for each part individually. This also reduces the effort of troubleshooting, because it allows narrowing down the number of faulty parts.