V2 is soon

[AqlaSolutions]

These features are already available in the V2 beta version:

1. Google Protocol Buffers format support is back (I've reconsidered this) - (de)serialization.
2. Improved reference tracking (e.g. surrogate fix and referencing arrays from inside themselves).
3. Improved versioning (e.g. between reference-nonreference-null-nonnull-dynamic).
4. Optimizations for better output size.
5. Nested collections.
6. Collection subtypes (will read subtype number to create correct concrete type).
7. Array types may be registered as collections and use full set of features (null support, etc) even when passed as root objects.
8. Primitive types are allowed to be set to fields marked as dynamic type.
9. Reference serialization will avoid using too much recursion Exception When Trying to Serialize a Bigger Graph Object #1.
10. Possibility to specify different attribute sets for different models (see ModelId property on attributes).
11. Attributes to specify behavior on collection and collection elements (and nested) separately.
12. Multi-dimensional arrays.

And those are planned to be added with minor 2.x releases:
12. No-versioning mode for best output size.
14. Google Protocol Buffers schema correct generation inside AqlaSerializer.
15. Protogen to generate classes from Google Protocol Buffers schema.
16. Improved performance by avoiding boxing when serializing Nullable[T] with null support or advanced versioning mode enabled.
17. CoreCLR platform support.

AppendCollection mode is not going to be supported in complex scenarios.

Migration:

Attributes mapping example:

[SerializableMember(5, ValueFormat.MinimalEnhancement)] // List<T>
[SerializableMemberNested(1, ValueFormat.Reference)] // int[]
[SerializableMemberNested(2, ValueFormat.Compact)] // int
public List<int[]> Member { get; set; }

[SerializableMemberNested] is optional and may be omitted.

Code mapping example:

MetaType mt = model[typeof(MyType)];
ValueMember valueMember = mt[5];

var s = valueMember.GetSettingsCopy(0); // List<T>
s.Format = ValueFormat.MinimalEnhancement;
valueMember.SetSettings(s, 0);

s = valueMember.GetSettingsCopy(1); // int[]
s.Format = ValueFormat.Reference;
valueMember.SetSettings(s, 1);

s = valueMember.GetSettingsCopy(2); // int
s.Format = ValueFormat.Compact;
valueMember.SetSettings(s, 2);

[ab-tools]

That is really great news!

Especially number 9 is very interesting for me as although the current version works with extended stack size it is quite slow.

When you think the first usable version will be ready?

The reason I'm asking this is that we will release the product where I'm currently using AqlaSerializer very extensively in about 2-3 weeks:
After this release I basically cannot change the serialization anymore as long as this is not backwards compatible for reading.

Or is there a chance that we will have V2 backwards compatibility - just for reading the V1 format?

[AqlaSolutions]

@ab-tools, unfortunately V2 won't be wire compatible with V1. In V2 the serialization flow is significantly changed.

Actually all the promised changes are already implemented but without compilation support yet. You may check the preview version:
https://github.com/AqlaSolutions/AqlaSerializer/releases/tag/v.dev-preview-v2

The emit (compilation) part is very large... I set the milestone to February 25.

[ab-tools]

First thanks for your quick reply as always! :-)

unfortunately V2 won't be wire compatible with V1. In V2 the serialization flow is significantly changed.

Yes, I guessed that already, but thought maybe you could implement a fallback like if it detects V1 format, it processes it like in V1 version (just read-only). But if V2 comes so quick, it's not needed for me, then I will use V2 directly from my first product release.

Actually all the promised changes are already implemented but without compilation support yet.

I don't need any compilation support, just a "stable" version that I can use:
The most important part for me to know is the point when there will be no more format changes. I mean that every V2 file I created can be read with upcoming V2 versions.

Would you consider the current pre-release version already "format fixed" or do you expect some format incompatibility changes till the release on February 25?

[AqlaSolutions]

@ab-tools

1. There will be a change in collection subtypes serialization soon. I'm not planning to change the format afterwise.
2. I still can't guarantee that there are no hidden bugs which may make me to change the format. But the tests are mostly green (except those relying on emit).
3. Attributes and default settings changes are planned and they may affect which settings are actually applied (e.g. null and reference support on members).
4. After the final release you may want to change some setttings (like AdvancedVersioning mode) for better performance/size which will may the format incompatible.

I would wait as long as possible before making the final decision about migrating to V2 format.

[ab-tools]

OK, good, I'll wait as long as possible, but also need to get the product release finally done (is already a bit late as always ;-) ).

It would be great if you could do a quick comment here when point 1 is done as it seems I at least need to wait for that.

Thanks again a lot
Andreas

P. S.:
If everything goes well with V2 and I really can used that for my product which would be according to your description above for me really a BIG step forward, I definitely need your PayPal e-mail address for a donation - I'm very thankful for all your efforts you put in this and want at least to give a little bit back!

[AqlaSolutions]

@ab-tools

Especially number 9 is very interesting for me as although the current version works with extended stack size it is quite slow.

I should warn you that the number 9 goal is not to speed things up but to avoid the necessity to create another thread with big stack size. It may be even slightly slower than the recursive version because it does more things like boxing, maintaining reference list, etc.. Although thread creation itself is a slow procedure and big stack size doesn't fit good in CPU cache so number 9 may still turn up faster for you.

Good news!

I've finished those changes to lists and now I'm not going to introduce any format changes for V2 release. Note that the API is planned to be changed soon. You may try to build the last version from the V2 branch. This https://github.com/AqlaSolutions/runsharp is included to the solution as a project reference so you'll have to clone it too.

Now its defaults are suitable only for testing so you need to disable some things for your real use model like this:

var comp = ProtoCompatibilitySettings.None;
comp.AllowExtensionDefinitions &= ~NetObjectExtensionTypes.AdvancedVersioning;
comp.AllowExtensionDefinitions &= ~NetObjectExtensionTypes.LateReference;
var model = TypeModel.Create(false, comp);

"LateReference" is number 9 but for now it's model-wide setting and can't be toggled per type. You'd want it only for some specific properties so better disable for now. The same is true for "AdvancedVersioning".

Null support is also enabled (where applicable) model-wide in this version. Attribute option "SupportsNull" doesn't change anything yet.

If you setup this way you should be able to read the data in V2. But if you change later those settings per type in V2 (e.g. enable LateReference for some members) you won't be able to read the old data so you'll need to setup two models - 1st for reading the old data and 2nd for using the new features. This gives me an idea of a feature where each attribute may have a "set-id" option allowing to auto-add the same field in different way to multiple models.

Also note that AqlaSerializer now supports plain Google Protocol Buffers format the same way like protobuf-net does (without referencing, null handling, etc) and it will be always possible to read it in any upcoming V2 versions. (protobuf-net specific features are not compatible, only plain Google Protocol Buffers)

[AqlaSolutions]

@ab-tools I made another format change today. I hope it's not too late. Please notify me when you release your app and it's really impossible to change the format anymore.

[ab-tools]

First I'm very sorry to not get back to you earlier - I really appreciate your great work and also keeping me up-to-date on the status.

In fact we're currently very busy to get everything ready for release, but we did not release yet. So the last format change was not a problem.

I will do some tests (especially regarding read performance which is important for us) today evening and will let you know the results.

Thanks again
Andreas

[ab-tools]

Just wanted to compile the latest V2 version, but I only have Visual Studio 2013 and it seems you are using a lot VS 2015 features now.

Could you therefore please provide the latest V2 version any binary?
Then I can do a test with them later today.

Thanks!
Andreas

[AqlaSolutions]

@ab-tools https://github.com/AqlaSolutions/AqlaSerializer/releases/tag/v.dev-preview-2-2-v2

[ab-tools]

Saw that build already, but it seems to be already 20 days old.

Doesn't it make more sense to test with your latest version?

[AqlaSolutions]

@ab-tools it's new, it's just somehow attached to the master branch dates but I'm working in V2 now.

[AqlaSolutions]

@ab-tools oops, now I know why ;)
I've just reuploaded it correctly.

[ab-tools]

Now it looks much better, thanks! :-)

[ab-tools]

Just wanted to do a first test with V2, but I cannot compile the project anymore after referencing the new DLL due to "missing System.Runtime.CompilerServices.ExtensionAttribute..ctor" compilation error.

Sorry, but looks like I need a .NET 4 version of the DLL to reference it in my project. I compile against .NET 4.0 x86.

[AqlaSolutions]

@ab-tools Oh, I see. Let me check it.

[AqlaSolutions]

@ab-tools I created a new .NET 4.0 project, referenced the dll, used it in Program.cs and I see no compiler errors there. I'm not sure what may cause your issue.

[AqlaSolutions]

@ab-tools please download the updated binaries from the same link and try to compile using them. I have a guess what may cause the error. But I still can't reproduce it.

[ab-tools]

I really just removed the old AqlaSerializer reference and added the new one. While trying to compile again afterwards, I got this compilation error.

I will try it with your binary again directly tomorrow morning, thanks!

[AqlaSolutions]

@ab-tools I uploaded the full build, you can use net35/net40 version.

[ab-tools]

Just wanted to let you know first that I still get the same compilation error also when I use the .NET 4 version. Maybe not both of the dependencies are .NET 4 as well?

Anyway, it works find with the work-around you mentioned on the download page.
So I can do some first tests now and let you know the results then.

[AqlaSolutions]

@ab-tools I'm absolutely sure (and I've just checked via reflector) that net40 dlls version doesn't include its own declaration of ExtensionAttribute (which was the cause). Either you are not referencing the correct versions of mscorlib and System.Core inside your project or you are still referencing that net30 version.

[ab-tools]

OK, I will double check it once more, maybe I just selected the wrong file after the first test (which is currently running) has finished.

[ab-tools]

I'm sorry, this time it was really my fault - just used the DLL from the wrong folder.
With the .NET 4.0 version everything compiles just fine now!

I also finished first test with the parameters

var comp = ProtoCompatibilitySettings.None;
comp.AllowExtensionDefinitions &= ~NetObjectExtensionTypes.AdvancedVersioning;
comp.AllowExtensionDefinitions &= ~NetObjectExtensionTypes.LateReference;
var model = TypeModel.Create(false, comp);

as you suggested above and here are my results:

• Saving and loading my 2 database files in the project works fine without any code change except using the options above.
• It seems to be a bit slower, but that's probably because "compilation support" is not enabled yet, right?
• Then I changed my code to no longer create a new thread with a bigger thread size, but load the database on the main thread: Result was that is throws again a StackOverflowException at me. I thought this problem would be gone now, isn't it?

Thanks again
Andreas

[AqlaSolutions]

@ab-tools

1. I'm glad to hear it :)
2. yes, with the compiler support it should be much faster than without.
3. if you remove this line

comp.AllowExtensionDefinitions &= ~NetObjectExtensionTypes.LateReference;

then it should work without new thread (can you check pls?) but it will write all the references using LateReference technique which is not optimal. I suggest you to keep it disabled until I add a possibility to specify LateReference mode per member. Also LateRefernce switch is not compatible between versions,

if you change later those settings per type in V2 (e.g. enable LateReference for some members) you won't be able to read the old data so you'll need to setup two models - 1st for reading the old data and 2nd for using the new features. This gives me an idea of a feature where each attribute may have a "set-id" option allowing to auto-add the same field in different way to multiple models.

[ab-tools]

First thanks for your very quick reply as always. :-)

1. (empty - stupid Markdown syntax)
2. Very good. Although it's usable at the moment, program start takes some seconds longer which is not so good as it's quite slow already anyway (due to the amount of data that needs to be loaded), but if that gets better with the "compilation support" again that's OK.
3. Just for my understanding "LateReference" is something different than the "ByReference" stuff which I can enable/disable type based by "NotAsReferenceDefault", right?

I did a test with "LateReference" enabled now and yes, now it works without a new thread!
But it seems to be again considerable slower then unfortunately and the resulting file size is about 16 % bigger (doesn't matter for me, just wanted to mention it - for me only speed is relevant, not size). By the way in the first test with "LateReference" disabled the file size was even a bit smaller than with V1, but only 2 % or so.

Also LateRefernce switch is not compatible between versions

As our product is a bit delayed now anyway and we still need to fix some remaining issues and get everything ready (manual and stuff like that) we postponed the official release now not till end of this month.

Although that's annoying for me it should fit very well to be after your official release of V2 and everything should be ready then including the "LateReference" switch. :-)

[AqlaSolutions]

@ab-tools 2 - there is NotAsReference on members which is also affected by NotAsReferenceDefault on a member type. LateReference may be used only for members which have reference tracking enabled. It changes reference tracking behavior to not use deep traversal ("to not go inside member recursively") but it has to write additional data and therefore it's slower. I recommend to use it only on concrete members which should not be traversed recursively.

It's called "LateReference" because it's processed later, outside of its container object.

[ab-tools]

@AqlaSolutions: OK, no problem, then I just keep my surrogate for now. Thanks!

[ab-tools]

@AqlaSolutions: After all my tests were successful I just switched to V2 for the latest beta version of my product. :-)

One thing I just wanted to mention/ask:
Do you think it would be possible to use "normal" LINQ when compiling against .NET 4.0 or newer instead of this "LinqBridge.cs"?

I just did this change as I compiled it myself anymore and this saved about 50 KB of DLL size. Although size is not so important these days it seems just redundant to include an "own" LINQ implementation instead of using the .NET one.

Maybe you could switch to "normal" LINQ for .NET 4.0 and newer via compiler directives?

[AqlaSolutions]

@ab-tools I decided to use the same LINQ implementation in all targets because different behaviors may cause unexpected and not reproducable bugs - even when it looks like simple thing in reality they behave differently in corner cases. It's a kind of overcautiousness but the size difference is not that big...

When you want to have minimal binaries size you usually run your product in limited .NET environment like mobile platforms where you would need to use CoreOnly version with precompiled model dll and without RuntimeTypeModel.

I will check whether I can remove LinqBridge from CoreOnly version completely and if it's not possible then I will go the way you asked because minimizing CoreOnly build is more important than having the same implementation.

Anyway this is something that may be chosen later when more important changes like V2 release are done.

[ab-tools]

@AqlaSolutions: OK, didn't expected that they may behave differently.

But as I'm only using the .NET 4 version (don't have other projects with different targets it need to talk to), it should not cause any problems to use native LINQ hopefully.

[aienabled]

@AqlaSolutions, is there any wiki page covering inheritance support?
AqlaSerializer is positioned as serializer for objects, not simple data. So I expect some improvements in how it handles hierarchies in comparison to protobuf-net.

For example, I have a class with complex hierarchy, and I need to be able to serialize it. I expect it to work out-of-box, but it (source from latest v2_attributes branch) doesn't serialize private fields of base classes. For network transmission this could be acceptable, but for database scenario it usually causes an unexpected data loss.

In case of protobuf-net, to serialize private fields from base classes, I need to add MetaType for all its base class chain and for each of them specify inheritors by calling AddSubType as discussed in this question on SO. That's quite hard even with code-only approach (without attributes).

So I would like to see a wiki page covering inheritance support in AqlaSerializer.

BTW, I found a typo: "Comparsion with protobuf net and migration" (Comparsion Comparison).

Regards!

[AqlaSolutions]

@aienabled, there is an automatic subtype registration in attribute mapping when both base and derived type are marked with [SerializableType]. In such scenario they should be registered in the same order to have the same ids. Please see also https://github.com/AqlaSolutions/AqlaSerializer/wiki/Batch-types-registration

When you register type from code you still have to specify subtypes manually. But If you use attributes you can simply specify [SerializableType(ImplicitFields = ImplicitMode.AllFieldsAndProperties)] to implicitly add all current type fields.

It's important to have subtypes registered with field numbers because this way each hierarchy has independent type key numbers and you may add different hierarchies in any order. Instead of writing concrete type index of type in Model.Types[] serializer writes subtype index which is local for each base type. Moreover, when a concrete type is not registered in another version deserializer will still read the most concrete possible type from its hierarchy.

Also see https://github.com/AqlaSolutions/AqlaSerializer/wiki/Changing-auto-add-behavior about ImplicitFallbackMode of DefaultAutoAddStrategy. ImplicitFallbackMode also supports automatic subtypes registration. You can setup ImplicitFallbackMode and add types from code with auto = true.

The current code-only API provides very low level access so you don't have anything "automatic" there. Right now I'm not planning to make a more high-level mapping API but when I finish with attributes you will have more extension points inside DefaultAutoAddStrategy. (But please don't use those extension points now because I'm going to alter that part for attributes.)

[aienabled]

@AqlaSolutions, thank you for detailed response! Please consider adding a special wiki page based on it - it could be very helpful for new developers.

I just checked and can confirm the attributes approach is working fine.
However, I found the same problem as with my code-oriented approach: exception The type cannot be changed once a serializer has been generated thrown when I try to serialize another type which uses the same base type (after serializing object of some type using the same base type). It happens because serializer for the base type is already generated (the test code if you're interested).

It seems the issue is by design... this is resolvable by cloning RuntimeTypeModel, but for developers using RuntimeTypeModel.Default is not a solution of the issue as Default is getter-only property. Maybe it would be better to add RuntimeTypeModel.Reset() (static method, for cloning Default instance). However I think it doesn't resolve the problem completely because it need to be called manually.

Going back to the my problem. I'm dynamically adding new types into the RuntimeTypeModel, and most of this types have exactly the same type names (yes, I need it to work this way to convert objects of "old" type to objects of "new" type - but type FullName exactly the same). I'm cloning/unfreezing the model before extending it with new types. Without inheritance it worked perfectly well (I've successfully used it for a few days), but with inheritance it cannot properly deserialize objects into the new types (Deserialize() returns value of old type). It seems to be expected behavior (protobuf scheme):

message BaseClass {
   optional uint32 id = 1 [default = 0];
   // the following represent sub-types; at most 1 should have a value
   optional DeferredClass DeferredClass = 128;
   optional DeferredClass DeferredClass = 129;
}

• so it cannot deserialize DeferredClass to the new type (fieldId=129) because it was serialized as field Id=128 and deserialized as old type (even if I provided new type to Deserizalize() method).

I'm stuck with it now and seeking for any ideas how I could avoid it this problem without creating separate RuntimeTypeModel. Please note I'm not using attributes and using code-only model configuration.

Is it possible to use something like "flatten hierarchy" when the serializer for the specific type itself contains logic for (de)serializing private fields of base class(es) - exactly the same way it (de)serialize self fields and properties? It will add some duplication to scheme, but it will completely resolve the problem I have because this way it's not required to add any extra fields for subtypes to the base type TypeModel (it even will not require adding base types into the model and cloning the model to add new types). Is that achievable with current architecture? If so, could you point me please on the extension points which I could use to add support for it?

Regards!

UPD. I just found a workaround by adding a method public ValueMember AddField(int fieldNumber, FieldInfo fieldInfo, Type itemType = null, Type defaultType = null, object defaultValue = null) which is basically uses the same code as current private ValueMember AddField(int fieldNumber, string memberName, Type itemType, Type defaultType, object defaultValue). With this method I can manually provide all the fields from base class(es) and it (de)serializes them correctly and without any issues. So I have working "flatten hierarchy" approach now.

[AqlaSolutions]

@aienabled

It seems the issue is by design... this is resolvable by cloning RuntimeTypeModel

Yes, I added a special API method CloneAsUnfrozen for such case. But better just use TypeModel.Create to make your own RuntimeTypeModel instead of RuntimeTypeModel.Default and no need to depend on it or Reset.

Is it possible to use something like "flatten hierarchy"

It's not hard to add such option. But it will be impossible to determine which subtype you serialized so you'll have to tell deserializer registered (not base) type. Such feature won't be top-priority for me in near future. You can add it by yourself, check the DefaultAutoAddStrategy code. It looks like that you just need to remove BindingFlags.DeclaredOnly from member search options and disable base type registration.

Possible solutions for your issue:

1. Make a reusable code to setup new TypeModel.Create-d model with either old types or new. Use it to create two models exactly the same way - 1 for serialization, 2 for deserialization.
2. Use different base classes for old and new hierarchies. Map them the same way. Pass new base type to Deserialize method. It will work only if you use different types for the most abstract registered type of the hierarchy.

[aienabled]

@AqlaSolutions

You can add it by yourself, check the DefaultAutoAddStrategy code. It looks like that you just need to remove BindingFlags.DeclaredOnly from member search options and disable base type registration.

Unfortunately, .NET Reflection will not return private fields of base class(es) if GetMember() is invoked for deferred class type (there is a question on SO). So my issue cannot be resolved this way.

1. Make a reusable code to setup new TypeModel.Create-d model with either old types or new. Use it to create two models exactly the same way - 1 for serialization, 2 for deserialization.

Yes, I'm also considered this sophisticated approach. However, it will require total refactoring of my architecture. Currently I'm using the same instance of RuntimeTypeModel for everything.

So it seems the easiest option for me is using fork of your code which allows adding FieldInfo directly (not by name) to field metadata as I described in UPD section of my previous message. It suits very well for my current needs.

Regards!

[AqlaSolutions]

@aienabled ok, cool then )

[AqlaSolutions]

Beta is out!

https://github.com/AqlaSolutions/AqlaSerializer/releases

I added mapping examples to the header in this issue.

@ab-tools
Please check the compatibility. You may need to add some attributes but the format is the same.

[ab-tools]

@AqlaSolutions: OK, I will give it a try, thanks!

But I'm not sure if I understood the new mapping attributes correctly yet:

1. Do I need to apply these to all collections (List, Dictionary, ...)?
2. What I need to apply for something like this: public List<CustomObject> CustomObjects { get; set; }?
3. What does the serializer uses by default if no such attributes are applied?

[AqlaSolutions]

@ab-tools

What does the serializer uses by default if no such attributes are applied?

Default comes from type settings, otherwise Reference if appropriate, otherwise MinimalEnhancement for nullables, otherwise Compact.

For collections default CollectionMode is Enhanced.

What I need to apply for something like this: public List CustomObjects { get; set; }

You may leave it by default unless you want some specific behavior. Just [SerializableMember(n)] is enough.

Do I need to apply these to all collections (List, Dictionary, ...)?

Again, only when you need some specific behavior for collection items.

If you had specified non-default settings on [SerializableMember] you will need to specify the same for all nested levels with [SerializableMemberNested(level)] as now they are not copied from [SerializableMember] anymore.

[ab-tools]

@AqlaSolutions: I meant what I need to apply for public List CustomObjects { get; set; } to keep it compatible with the pre-release version - I guess it's this then, right?

[SerializableMember(5, ValueFormat.MinimalEnhancement)] // List<T>
[SerializableMemberNested(1, ValueFormat.Reference)] // CustomObjects
public List<CustomObject> CustomObjects { get; set; }

And it would be great if you could write a short explanation (as I guess the Wiki is not ready yet) what the different modes are all about, especially what the mode Enhanced does?

Thank you!

[AqlaSolutions]

@ab-tools such explanation is already present in xmldocs, use your IDE.

1/ Suppose you had

[SerializableMember(5)]
public List<MyObject> CustomObjects { get; set; }

You don't need to change anything here. The defaults are the same.

2/ Suppose you had

[SerializableMember(5, EnhancedFormat.Minimal, WriteAsDynamicType = true)]
public List<MyObject> CustomObjects { get; set; }

It should become now

[SerializableMember(5, ValueFormat.MinimalEnhancement, WriteAsDynamicType = true)] // List<T>
[SerializableMemberNested(1, ValueFormat.MinimalEnhancement, WriteAsDynamicType = true)] )] // CustomObjects
public List<MyObject> CustomObjects { get; set; }

* edited from LateReference as it doesn't support non-default settings on a current and nested levels.

[AqlaSolutions]

I added a wiki page which describes possible member formats.

Also collection formats.

Versioning and field numbers

@ab-tools

[AqlaSolutions]

@ab-tools you'll also need to set ((AutoAddStrategy)model.AutoAddStrategy).UseLegacyTupleFields = true before adding any types.

I want to ensure that everything is ok with format compatibility so I can publish the final V2 release without worrying. Please check the last build.

[ab-tools]

@AqlaSolutions: I'm really sorry that it takes so long, but we've just a lot of work right now directly after the release of our product. :-)

By the way, as it's released now, I can also share a link here if you would like to know where your great library is used: It's a instructor station for Prepar3D and FSX flight simulators.

I'll try to give your latest version a try today evening and will post the results then!

[AqlaSolutions]

@ab-tools Interesting, I downloaded your app, it looks like it doesn't detect trial FSX but loads database when I specify directories manually. So it's like a realtime mission/map editor, right?

Please check the format compatibility asap (if you still need it) because it might be impossible to change after the final release of V2 and I have everything ready for the release now.

If you see any problems try to prepare small examples of incompatible cases. Also you can use model.GetDebugSchema method to retrieve actual format settings.

[ab-tools]

@AqlaSolutions: Again sorry for the delay, I just started to test already and will send you feedback definitely today (probably in the next few hours already).

By the way, where you downloaded FSX trial from?
Normally it should detect that as well.

[ab-tools]

@AqlaSolutions: I just gave it a try now and only changed two occurances of EnhancedFormat.LateBinding to ValueFormat.LateBinding.

When I run that now, I get following exception:

Internal error; a missing key occurred
   at AqlaSerializer.NetObjectCache.GetKeyedObject(Int32 key) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\NetObjectCache.cs:line 38
   at AqlaSerializer.NetObjectHelpers.ReadNetObject_Start(Object& value, ProtoReader source, Type& type, NetObjectOptions options, Boolean& isDynamic, Boolean& isLateReference, Int32& typeKey, Int32& newObjectKey, Int32& newTypeRefKey, Boolean& shouldEnd) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\NetObjectHelpers.Read.cs:line 53
   at proto_23(Object , ProtoReader )
   at AqlaSerializer.Serializers.CompiledSerializer.AqlaSerializer.Serializers.IProtoSerializer.Read(Object value, ProtoReader source) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Serializers\CompiledSerializer.cs:line 71
   at AqlaSerializer.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object value, ProtoReader source, Boolean isRoot) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Meta\RuntimeTypeModel.cs:line 1098
   at AqlaSerializer.ProtoReader.ReadTypedObject(Object value, Int32 key, ProtoReader reader, Type type) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\ProtoReader.cs:line 647
   at AqlaSerializer.ProtoReader.ReadObject(Object value, Int32 key, ProtoReader reader) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\ProtoReader.cs:line 633
   at proto_58(Object , ProtoReader )
   at AqlaSerializer.Serializers.CompiledSerializer.AqlaSerializer.Serializers.IProtoSerializer.Read(Object value, ProtoReader source) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Serializers\CompiledSerializer.cs:line 71
   at AqlaSerializer.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object value, ProtoReader source, Boolean isRoot) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Meta\RuntimeTypeModel.cs:line 1098
   at AqlaSerializer.ProtoReader.ReadTypedObject(Object value, Int32 key, ProtoReader reader, Type type) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\ProtoReader.cs:line 647
   at AqlaSerializer.ProtoReader.ReadObject(Object value, Int32 key, ProtoReader reader) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\ProtoReader.cs:line 633
   at proto_4(Object , ProtoReader )
   at AqlaSerializer.Serializers.CompiledSerializer.AqlaSerializer.Serializers.IProtoSerializer.Read(Object value, ProtoReader source) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Serializers\CompiledSerializer.cs:line 71
   at AqlaSerializer.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object value, ProtoReader source, Boolean isRoot) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Meta\RuntimeTypeModel.cs:line 1098
   at AqlaSerializer.ProtoReader.ReadTypedObject(Object value, Int32 key, ProtoReader reader, Type type) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\ProtoReader.cs:line 647
   at AqlaSerializer.ProtoReader.ReadObject(Object value, Int32 key, ProtoReader reader) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\ProtoReader.cs:line 633
   at proto_252(Object , ProtoReader )
   at AqlaSerializer.Serializers.CompiledSerializer.AqlaSerializer.Serializers.IProtoSerializer.Read(Object value, ProtoReader source) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Serializers\CompiledSerializer.cs:line 71
   at AqlaSerializer.Meta.RuntimeTypeModel.Deserialize(Int32 key, Object value, ProtoReader source, Boolean isRoot) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Meta\RuntimeTypeModel.cs:line 1098
   at AqlaSerializer.Meta.TypeModel.DeserializeCore(ProtoReader reader, Type type, Object value, Boolean noAutoCreate, Boolean isRoot) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Meta\TypeModel.cs:line 793
   at AqlaSerializer.Meta.TypeModel.Deserialize(Stream source, Object value, Type type, Int32 length, SerializationContext context) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Meta\TypeModel.cs:line 752
   at AqlaSerializer.Serializer.Deserialize[T](Stream source) in D:\My Documents\Aqla\AqlaSerializer\src\protobuf-net\Serializer.cs:line 111
   at FlightControl.Program.loadSettings(Boolean retry) in C:\Austausch\#FS\FS-Control\Program.cs:line 557

I did not use something like EnhancedFormat.Minimal, so I'm not sure what I need to change additionally to make it compatible again.

Do you have any idea what I should search for?

Thanks
Andreas

[AqlaSolutions]

@ab-tools I used FSX from https://www.microsoft.com/Products/Games/FSInsider/downloads/Pages/FlightSimulatorXTrialVersion.aspx installed to a non-system drive.

Try to compare debug schemes with the latest AqlaSerializer and with alpha ( I ported debug schema over alpha: https://www.dropbox.com/s/hm9h7r9gxah0p3r/aqlaserializer%20alpha-r90.zip?dl=1 ).

model.GetDebugSchema(typeof(Database<IDatabase>);

Changing between Reference and MinimalEnhancement is ok.
Changing between Reference and LateReference is ok.
WithNullWireType format can read the alpha format (but not in opposite).
In collections PackedRead doesn't have an effect when combined with NewPacked.

I replaced attributes in the model you emailed me before and according to debug schemes everything should be compatible there if you have the same mapping as I am.

Feel free to send me the schemes if you need a help.

[ab-tools]

@AqlaSolutions: Just generated debug schemas for all objects that are used with AqlaSerializer for comparison and there are several differences.

Mainly WithNullWireType and AsReference are sometimes present, sometimes now.

Would be great if you could have a look - I sent them to you via e-mail.

Best regards and thanks a lot
Andreas

[AqlaSolutions]

@ab-tools

Are you using the latest release?

Map - should be compatible.

Settings and Database:

Seems like you've forgotten to set

((AutoAddStrategy)model.AutoAddStrategy).UseLegacyTupleFields = true 

before adding any types.

Field #2 of tuple KeyValuePair[String,Dictionary[String,String]]: is written with NetObject in release. Same for KeyValuePair[String,Dictionary[String,Single]] and several other tuples.

[ab-tools]

@AqlaSolutions: Oh, yes, you're right - I just forgot the UseLegacyTupleFields setting.

After this was added, it reads the old data perfectly fine!

By the way, is it possible to enable this lagacy tuple field mode only for reading?
Just asking as I'm only using AqlaSerializer. This way it only needs to read the "old" format once, but starting from there it could write with the "new" format without problems.

Best regards
Andreas

[AqlaSolutions]

@ab-tools you can use two models - with and without UseLegacyTupleFields. Just make sure you setup UseLegacyTupleFields before any types are added to model.

[ab-tools]

@AqlaSolutions: OK, I just left it as it is now with UseLegacyTupleFields as it works very well - reading and writing - in my further tests. So no problems here with the new version, thanks! :-)

By the way, we checked and this quite old demo version of FSX is indeed not fully compatible, but with the latest program version from our website you can at least build the database (=> use AqlaSerializer ;-) ) from the FSX Demo, but connecting to the flight simulator afterwards won't be possible.