Support VFP Varchar field

[kirides]

Varchars need special handling

Varchar can be up to 254 characters (always stored with constant size)
To retrieve the size of the Varchar

• you need to check the _NullFlags column of the Table (it's a hidden system column)
  (tables that contain a varchar always contain the _NullFlags column)
• depending on, if the null-Flag for the varchar field is set
  • (set) length is stored in the last byte of the field,
    if this is 0 the value should be interpreted as NULL
  • (not set) length equals the maximum allowed field length

_NullFlags-Column can be length-1 or more and represents a bitmap of all fields that can be nullable

[SebastiaanKlippert]

Thanks for the detailed description.

Do you have a table with this field type and some rows?
That would be easier to build and test it.

[kirides]

Created a table with 3 rows

varchar_table.zip

[SebastiaanKlippert]

Thanks, will try later.

[SebastiaanKlippert]

Trying out some things in https://github.com/SebastiaanKlippert/go-foxpro-dbf/tree/feature/varchars

The functionality you described is implemented in a basic rough form, but it is not trivial.
I did not take the _NullFlags field into account when designing this package. Need to think on it some more.

Current status:
In test TestVarChar the field values are now returned correctly, but it requires some more casting to check the value.
The returned type is *string, because the value can be nil, that is not pretty so a helper function may be required.

It also needs some code cleanup and it introduces some problems:

1. Field _NullFlags is now returned as a field in methods like Fields, NumFields, RecordToMap etc., this should be hidden I guess, but not sure if I can do that everywhere, would require some checking for the name at various places in the code.

2. http://foxcentral.net/microsoft/WhatsNewInVFP9_Chapter09.htm says:

If a field is both nullable and Varchar or Varbinary, two bits are used to represent a field. The lower bit represents the “full” status and the higher bit represents the null status. For example, a nullable 10-byte Varchar field containing “AB” is represented by 01 in _NullFlags (0 means not null, 1 means not full-size) while a null value in the same field is represented by 11 (null and not full-size).

This is currently not implemented, your table does not seem to have this, or I may be reading it wrong, because I don't know how to check it yet.

[SebastiaanKlippert]

You can check the output here if you don't want tot build it.

[kirides]

If a field is both nullable and Varchar or Varbinary, two bits are used to represent a field. The lower bit represents the “full” status and the higher bit represents the null status. For example, a nullable 10-byte Varchar field containing “AB” is represented by 01 in _NullFlags (0 means not null, 1 means not full-size) while a null value in the same field is represented by 11 (null and not full-size).

For some reason i did not find this paragraph in my researches, but yeah. After some testing, it is indeed like that, i implemented this change (two bits for a varchar) into my C# library.

here is a *.dbf with

• a nullable varchar row which value is NULL (row 1, Col 3, Nullable)
• a nullable varchar row which value is a empty string (row 2, Col 3, Nullable)
• a varchar row which which value is a full string (row 3, V (255), not null)

varchar_table.zip

[SebastiaanKlippert]

Thanks a lot, will continue working on this later.