Single-byte character set

[donmendelson]

In SBE the default encoding for single char and fixed-length character array was specified as US-ASCII. The motivation was to be compatible with FIX tagvalue encoding. When FIX started over 20 years ago, it was simply specified as ASCII character set, originally a 7-bit encoding. However, some FIX engines now support 8-bit "extended ASCII". Exactly what that means must be defined carefully because there are several related but slightly incompatible encoding standards. Some have interpreted this as ISO/IEC 8859-1. There is also a later 8859 part 15. Some discussion involves UTF-8, but that strikes me as wrong for this usage because it is a multi-byte character set, even though the first page is like ASCII.

Within the FIX standard, this will be specified more precisely for future usage. Meanwhile, I would like to solicit feedback on how this should be handled in SBE.

[JohnGreenan]

"Within the FIX standard, this will be specified more precisely for future usage."
On the old forum (not the current mini-orange one, not the terrible one that no-one used, but the one that was in use from the late 90s to say late 2009 there is a thread I started about character sets in old school tag=value FIX. I was seeing a problem that a FIX message was being created by a vendor supplied OMS & FIX engine that included in tag 58 a free format text field. I was working for a French firm at the time so comments often included French characters such as accents and cedilla. I could not see how a checksum could be calculated that was FIX compliant, hence I asked.

Jim Northey replied and had a very clear grasp of the issue for tag=value FIX.

For SBE I see two use cases:
(1) "Fast" - low-latency, high-frequency
(2) "Slow" - everything else.

For "fast" I see no real need for fancy character sets with double bytes. For "slow" I do. As an example, at the moment I am working with an SBE-FIXP-SOFH solution that will be used in South Korea, Japan, China and a number of other countries. So they would want to be able to transmit free text in their own language.

Therefore I think that the encoding standard MUST be inclusive, but can be dropped down to the bare bones where required.

On another note - will you be at the London conference? https://www.fixtrading.org/emea2019agenda/

I am speaking with Jim on Orchestra.

[donmendelson]

I understand that some FIX engines accept 8 bit encodings such as ISO 8859-1 but with ability to override by agreement between counterparties.

To clarify, there is already an attribute in the SBE schema to override encoding for fixed-length character arrays and variable-length fields:

US-ASCII is the default encoding of character arrays to conform to usual FIX values. The characterEncoding attribute may be specified to override encoding.

However, the default is specified as US-ASCII for fixed-length arrays. So the issue is to switch to an 8 bit encoding by default. A single byte character set is conducive to fixed-length arrays since characters==bytes. It would be possible to support multi-byte character sets in a byte array but perhaps there is a danger of buffer overrun in a translation.

[donmendelson]

If this proposal is accepted, examples with dumps should be updated to give precise character set rather than ASCII, i.e. ISO-8859-1.

[gelupa]

I understand that some FIX engines accept 8 bit encodings such as ISO 8859-1 but with ability to override by agreement between counterparties.

To clarify, there is already an attribute in the SBE schema to override encoding for fixed-length character arrays and variable-length fields:

US-ASCII is the default encoding of character arrays to conform to usual FIX values. The characterEncoding attribute may be specified to override encoding.

However, the default is specified as US-ASCII for fixed-length arrays. So the issue is to switch to an 8 bit encoding by default. A single byte character set is conducive to fixed-length arrays since characters==bytes. It would be possible to support multi-byte character sets in a byte array but perhaps there is a danger of buffer overrun in a translation.

For countries such as China or Japan, even a 8 bit encoding by default is not of much help. So the current design is OK, that is, the current default setting is US-ASCII , and give the option of using other encoding when needed.

Why will there be a buffer overrun when translation? In the current version of SBE, the length field is in octets, not in characters. It is not a problem now.

[donmendelson]

@gelupa, I was thinking about transcoding from a single-byte character set to multi-byte. But if your original text is in Unicode anyway, that case would not apply.