Skip to content

Alignment of var data and repeating groups #65 #67

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
donmendelson merged 3 commits into from
Feb 14, 2018
Merged

Alignment of var data and repeating groups #65 #67

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ab512fa
Alignment of var data and repeating groups #65
donmendelson Jan 8, 2018
caf4f9f
Alignment of var data and repeating groups #65
donmendelson Jan 8, 2018
f1878d2
#65 fix typo
donmendelson Jan 10, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion v2-0-RC1/doc/01Introduction.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,4 +185,4 @@ Representation of dates and times
Securities and related financial instruments - Codes for exchanges and
market identification (MIC)

XML 1.1 schema standards are located here [W3C XML Schema](http://www.w3.org/XML/Schema.html#dev)
[W3C XML Schema version 1.1](https://www.w3.org/TR/2006/REC-xml11-20060816/)
53 changes: 42 additions & 11 deletions v2-0-RC1/doc/03MessageStructure.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Message Structure
Message Framing
------------------------------------------------------------------------------------------------------------------------------------------------------------

SBE messages have no defined message delimiter. Version 2.0 of SBE makes it possible to walk the elements of a message to determine its limit, even when the message has been extended. Nevertheless, since internal framing depends on a correct starting point and not encountering malformed messages, it may be desirable to use an external framing protocol when used with transports that do not preserve message boundaries, such as when they are transmitted on a streaming
SBE messages have no defined message delimiter. SBE makes it possible to walk the elements of a message to determine its limit, even when the message has been extended. Nevertheless, since internal framing depends on a correct starting point and not encountering malformed messages, it may be desirable to use an external framing protocol when used with transports that do not preserve message boundaries, such as when they are transmitted on a streaming
session protocol or when persisting messages in storage.

### Simple Open Framing Header
Expand Down Expand Up @@ -78,7 +78,7 @@ The fields of the SBE message header are:
Block length is specified in a message schema, but it is also serialized
on the wire. By default, block length is set to the sum of the sizes of
body fields in the message. However, it may be increased to force
padding at the end of block. See section 3.3.3.3 below.
padding at the end of block. See [Padding](#padding-at-end-of-a-message-or-group).

### Message header schema

Expand All @@ -87,7 +87,7 @@ position as shown below. Each of these fields must be encoded as an
unsigned integer type. The encoding must carry the name "messageHeader".

The message header is encoded in the same byte order as the message
body, as specified in a message schema. See section 4.3.1.
body, as specified in a message schema. See section 4.

Recommended message header encoding

Expand Down Expand Up @@ -120,9 +120,9 @@ for blockLength.

The total space reserved for the root level of the message not counting
any repeating groups or variable-length fields. (Repeating groups have
their own block length; see section 3.4 below. Length of a
their own block length; see [Group block length](#group-block-length) below. Length of a
variable-length Data field is given by its corresponding Length field;
see section 2.7.3 above.) Block length only represents message body
see section 2.) Block length only represents message body
fields; it does not include the length of the message header itself,
which is a fixed size.

Expand All @@ -134,18 +134,18 @@ schema.

### Template ID

The identifier of a message type in a message schema. See section 4.5.2
below for schema attributes of a message.
The identifier of a message type in a message schema. See section 4
for schema attributes of a message.

### Schema ID

The identifier of a message schema. See section 4.3.1 below for schema
The identifier of a message schema. See section 4 for schema
attributes.

### Schema version

The version number of the message schema that was used to encode a
message. See section 4.3.1 below for schema attributes.
message. See section 4 for schema attributes.

### Number of repeating groups

Expand Down Expand Up @@ -227,6 +227,8 @@ would cause fields to overlap.
Extra octets specified for padding should never be interpreted as
business data. They should be filled with binary zeros.

The offset attribute is mutually exclusive with alignment; see below.

Example of fields with specified offsets

```xml
Expand All @@ -247,6 +249,29 @@ Example of fields with specified offsets
| OrderQty | 4 | 1 | 16 |
| Symbol | 8 | 0 | 20 |

#### Field byte alignment
As an alternative to specifying a field's offset from the start of a block, its byte alignment may be specified. If specified, the start of a field should reside at an offset from the start of the buffer that is the next multiple of the alignment attribute. If the offset without alignment would not be a multiple of the alignment attribute, then the buffer is padded with octets to that multiple. The padding should be set to binary zeroes.

The alignment attribute is mutually exclusive with offset.

If neither offset nor alignment attributes are set, then a field is positioned at the next available octet in a buffer without padding.

Computation of padding and effective offset is as follows, where effective-offset is the resulting position in the buffer:

```
padding = (alignment - (unaligned-offset modulo alignment)) modulo alignment
effective-offset = unaligned-offset + padding
```

Example of fields with alignment that is equivalent to the offset example above:

```xml
<field name="ClOrdID" id="11" type="string14" semanticType="String"/>
<field name="Side" id="54" type="char" semanticType="char"/>
<field name="OrderQty" id="38" type="intQty32" alignment="4" semanticType="Qty"/>
<field name="Symbol" id="55" type="string8" alignment="4" semanticType="String"/>
```

#### Padding at end of a message or group

In order to force messages or groups to align on byte boundaries or map
Expand Down Expand Up @@ -308,6 +333,12 @@ fields. (Length of a variable-length Data field is given by its corresponding
Length field.) Block length only represents message body fields; it does not
include the length of the group dimension itself, which is a fixed size.

### Byte alignment of a group entry

If byte alignment of a repeating group is specified, the start of the first field of each entry should reside at an offset from the start of the buffer that is the next multiple of the alignment attribute. If the offset without alignment would not be a multiple of the alignment attribute, then the buffer is padded with octets to that multiple. The padding should be set to binary zeroes.

If byte alignment is not specified for a group, then no padding is introduced to align entries.

### Padding at end of a group entry

By default, the space reserved for an entry is the sum of a group’s
Expand All @@ -322,7 +353,7 @@ each entry and should be set to zeroes by encoders. The blockLength
value does not include the group dimensions itself.

Note that padding will only result in deterministic alignment if the
repeating group contains no variable-length fields.
repeating group contains no variable-length fields. Therefore, the alignment attribute is a more direct solution to that need.

### Entry counter

Expand Down Expand Up @@ -507,7 +538,7 @@ finally, variable-length data fields.
Message structure validation
--------------------------------------------------------------------------------------------------------------------------

Aside from message schema validations (see section 4.8 below), these
Aside from message schema validations (see section 4), these
validations apply to message structure.

If a message structure violation is detected on a received message, the
Expand Down
12 changes: 10 additions & 2 deletions v2-0-RC1/doc/04MessageSchema.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ XML schema for SBE message schemas
---------
See [sbe.xsd](../resources/sbe.xsd) for the normative XML Schema Definition (XSD) for SBE.

The SBE schema is defined with W3C XML Schema Definition Language (XSD) version 1.1.


XML namespace
-----------------------------------------------------------------------------------------------------------
Expand Down Expand Up @@ -379,6 +381,10 @@ The `name` and `id` attributes are required. The first is a display name for
a message, while the latter is a unique numeric identifier, commonly
called template ID.

### Byte alignment

By default, a message starts at the first available byte of buffer. However, if desired, it may be aligned to a boundary by specifying the `alignment` attribute to improve performance on certain platforms. If specified, the start of the message should reside in the buffer at the next multiple of the alignment attribute. For example, if `alignment=4` then the message should start at the next address that is a multiple of 4.

### Reserved space

By default, message size is the sum of its field lengths. However, a
Expand Down Expand Up @@ -415,6 +421,7 @@ that they are encoded on the wire.
| semanticType | Documents value of FIX MsgType for a message | token | optional | Listed in FIX specifications |
| sinceVersion | Documents the version of a schema in which a message was added | nonNegativeInteger | default = 0 | |
| deprecated | Documents the version of a schema in which a message was deprecated. It should no longer be sent but is documented for back-compatibility. | nonnegativeInteger | optional | Must be less than or equal to the version of the message schema. |
| alignment | Controls byte alignment of the start of a message in its buffer| positiveInteger | optional | |

Note that there need not be a one-to-one relationship between message
template (identified by `id` attribute) and `semanticType` attribute. You
Expand Down Expand Up @@ -442,7 +449,8 @@ These are the common attributes of all field types.
| id | Unique field identifier (FIX tag) | unsignedShort | required | |
| description | Documentation | string | optional | |
| type | Encoding type name, one of simple type, composite type or enumeration. | string | required | Must match the name attribute of a simple `<type>`, `<composite>` encoding type, `<enum>` or `<set>`. |
| offset | Offset to the start of the field within a message or repeating group entry. By default, the offset is the sum of preceding field sizes, but it may be increased to effect byte alignment. | unsignedInt | optional | Must be greater than or equal to the sum of preceding field sizes. |
| offset | Offset to the start of the field within a message or repeating group entry. By default, the offset is the sum of preceding field sizes, but it may be increased to effect byte alignment. | unsignedInt | optional | Must be greater than or equal to the sum of preceding field sizes. Mutually exclusive with alignment |
| alignment | Controls byte alignment of the start of a field in its buffer| positiveInteger | optional | Mutually exclusive with offset |
| presence | Field presence | enumeration | Default = required | required = field value is required; not tested for null. optional = field value may be null. constant = constant value not sent on wire. |
| valueRef | Constant value of a field as a valid value of an enumeration | qualifiedName\_t | optional Valid only if presence= ”constant” | If provided, the qualified name must match the name attribute of a `<validValue>` within an `<enum>` |
| sinceVersion | The version of a message schema in which this field was added. | InonnegativeInteger | default=0 | Must not be greater than version attribute of `<messageSchema>` element. |
Expand Down Expand Up @@ -487,7 +495,7 @@ The number of members of each type is unbound.
| id | Unique group identifier | unsignedShort | required | |
| description | Documentation | string | optional | |
| dimensionType | Dimensions of the repeating group | symbolicName\_t | default = groupSizeEncoding | If specified, must be greater than or equal to the sum of field lengths. |

| alignment | Controls byte alignment of the start of each group instance in its buffer| positiveInteger | optional | |
`<group>` element inherits attributes of blockType. See `<message>`
above.

Expand Down
2 changes: 2 additions & 0 deletions v2-0-RC1/doc/05SchemaExtensionMechanism.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ Compatibility is only ensured under these conditions:

- Existing fields cannot change data type or move within a message.

- Message and repeating group byte alignment may not change.

- A repeating group may be added after existing groups at the root level or nested within another repeating group.

- A variable-length data field may be added after existing variable-length data at the root level or within a repeating group.
Expand Down
Loading