New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invalid wire-type exception after upgrading to 2.2.0 #256

Closed
mburbea opened this Issue May 25, 2017 · 22 comments

Comments

Projects
None yet
3 participants
@mburbea

mburbea commented May 25, 2017

I have a very large list of a type with many fields, that is stored in a zip archive file. The file was saved with version 2.1.0

When I try to read the archive file in 2.2.0 I get an invalid wire format exception complaining about serializing datetimes. This error does not happen in 2.1.0

The error happens on the first element in the list.

Unhandled Exception: ProtoBuf.ProtoException: Invalid wire-type; this usually me
ans you have over-written a file without truncating or setting the length; 
see http://stackoverflow.com/q/2152978/23354
   at ProtoBuf.BclHelpers.ReadTimeSpanTicks(ProtoReader source, DateTimeKind& kind)
   at ProtoBuf.BclHelpers.ReadDateTime(ProtoReader source)

I have tried producing a minimal sample, by creating a proto object which only has those datetime fields. But I can not get this error.

The first field in my object is a datetime so I suspect this is related. There are gaps in my objects proto numbering but I'm not sure if that matters. The root object has 14 fields.

The error happens when there are at least 2 objects in the list. I can send the zip and the code to read it, if you'd like to take a look.

Minimal code sample that is causing the error::

using (var f = new FileStream("dummy.zip", FileMode.OpenOrCreate))
using (var zip = new ZipArchive(f, ZipArchiveMode.Read))
using (var s = zip.Entries.FirstOrDefault().Open())
{
     foreach (var o in Serializer.DeserializeItems<Pnl<Holding>>(s, PrefixStyle.Base128, 1))
     {
        Console.WriteLine(o); // error happens in iteration. Nothing ever got printed.
     }
}

Edit:
If I comment out the first field (the datetime), I get a different error.

Unhandled Exception: System.IO.EndOfStreamException: Attempted to read past the
end of the stream.
   at ProtoBuf.ProtoReader.SkipField()
@mgravell

This comment has been minimized.

Show comment
Hide comment
@mgravell

mgravell May 25, 2017

Owner

oh wow, that's worrying; does dummy.zip exist and have data in this minimal example? trying to repro locally...

Owner

mgravell commented May 25, 2017

oh wow, that's worrying; does dummy.zip exist and have data in this minimal example? trying to repro locally...

@mgravell

This comment has been minimized.

Show comment
Hide comment
@mgravell

mgravell May 25, 2017

Owner

k, can't repro with that code, because a: I don't have your Pnl<T> or Holding types, and b: I don't have your data, and if I do anything with an empty zip, the ZipArchive code exploded long before it gets near protobuf-net.

Owner

mgravell commented May 25, 2017

k, can't repro with that code, because a: I don't have your Pnl<T> or Holding types, and b: I don't have your data, and if I do anything with an empty zip, the ZipArchive code exploded long before it gets near protobuf-net.

@mgravell

This comment has been minimized.

Show comment
Hide comment
@mgravell

mgravell May 25, 2017

Owner

additional question: how large is the data when uncompressed?

Owner

mgravell commented May 25, 2017

additional question: how large is the data when uncompressed?

@mgravell

This comment has been minimized.

Show comment
Hide comment
@mgravell

mgravell May 25, 2017

Owner

The error happens when there are at least 2 objects in the list. I can send the zip and the code to read it, if you'd like to take a look.

Yes please, very much. Sorry for drip-feeding comments here!

Owner

mgravell commented May 25, 2017

The error happens when there are at least 2 objects in the list. I can send the zip and the code to read it, if you'd like to take a look.

Yes please, very much. Sorry for drip-feeding comments here!

@mburbea

This comment has been minimized.

Show comment
Hide comment
@mburbea

mburbea May 25, 2017

mburbea commented May 25, 2017

@mgravell

This comment has been minimized.

Show comment
Hide comment
@mburbea

This comment has been minimized.

Show comment
Hide comment
@mburbea

mburbea May 25, 2017

mburbea commented May 25, 2017

@mgravell

This comment has been minimized.

Show comment
Hide comment
@mgravell

mgravell May 27, 2017

Owner

Thank you, your email was perfect. I have isolated the problem (I know what is happening, where, and why), and a fix should be available this weekend. Good bug report, thanks.

https://twitter.com/marcgravell/status/868397653939429376

Owner

mgravell commented May 27, 2017

Thank you, your email was perfect. I have isolated the problem (I know what is happening, where, and why), and a fix should be available this weekend. Good bug report, thanks.

https://twitter.com/marcgravell/status/868397653939429376

mgravell added a commit that referenced this issue May 27, 2017

@mgravell

This comment has been minimized.

Show comment
Hide comment

@mgravell mgravell closed this May 27, 2017

@mburbea

This comment has been minimized.

Show comment
Hide comment
@mburbea

mburbea May 30, 2017

Thank you! Seems to be working great now.

mburbea commented May 30, 2017

Thank you! Seems to be working great now.

@newton3

This comment has been minimized.

Show comment
Hide comment
@newton3

newton3 Aug 1, 2017

I have exactly the same issue after upgrading from 2.1.0 to 2.3.0/2.3.1.
I do not get this error when using 2.2.1 or 2.1.0

@mgravell seems like the issue was introduced back with 2.3.0 and persists in 2.3.1. Please let me know if mistaken.

EDIT:
More details - error occurs while de-serializing the data using 2.3.0 or 2.3.1, which was serialized by using 2.1.0. Exception below -

Invalid wire-type; this usually means you have over-written a file without truncating or setting the length; see http://stackoverflow.com/q/2152978/23354
at ProtoBuf.ProtoReader.StartSubItem(ProtoReader reader)
at ProtoBuf.ProtoReader.ReadTypedObject(Object value, Int32 key, ProtoReader reader, Type type)
at proto_78(Object , ProtoReader )
at ProtoBuf.Meta.TypeModel.DeserializeCore(ProtoReader reader, Type type, Object value, Boolean noAutoCreate)
at ProtoBuf.Meta.TypeModel.Deserialize(Stream source, Object value, Type type, SerializationContext context)
at ProtoBuf.Serializer.Deserialize[T](Stream source)

newton3 commented Aug 1, 2017

I have exactly the same issue after upgrading from 2.1.0 to 2.3.0/2.3.1.
I do not get this error when using 2.2.1 or 2.1.0

@mgravell seems like the issue was introduced back with 2.3.0 and persists in 2.3.1. Please let me know if mistaken.

EDIT:
More details - error occurs while de-serializing the data using 2.3.0 or 2.3.1, which was serialized by using 2.1.0. Exception below -

Invalid wire-type; this usually means you have over-written a file without truncating or setting the length; see http://stackoverflow.com/q/2152978/23354
at ProtoBuf.ProtoReader.StartSubItem(ProtoReader reader)
at ProtoBuf.ProtoReader.ReadTypedObject(Object value, Int32 key, ProtoReader reader, Type type)
at proto_78(Object , ProtoReader )
at ProtoBuf.Meta.TypeModel.DeserializeCore(ProtoReader reader, Type type, Object value, Boolean noAutoCreate)
at ProtoBuf.Meta.TypeModel.Deserialize(Stream source, Object value, Type type, SerializationContext context)
at ProtoBuf.Serializer.Deserialize[T](Stream source)
@mgravell

This comment has been minimized.

Show comment
Hide comment
@mgravell

mgravell Aug 1, 2017

Owner

@newton3 do you have anything I can use to repro this? It is quite hard to diagnose just from a stacktrace - a minimal repro would make it a lot easier to find what the problem is

Owner

mgravell commented Aug 1, 2017

@newton3 do you have anything I can use to repro this? It is quite hard to diagnose just from a stacktrace - a minimal repro would make it a lot easier to find what the problem is

@newton3

This comment has been minimized.

Show comment
Hide comment
@newton3

newton3 Aug 1, 2017

I understand. Trying to come up with a minimal repro.

newton3 commented Aug 1, 2017

I understand. Trying to come up with a minimal repro.

@newton3

This comment has been minimized.

Show comment
Hide comment
@newton3

newton3 Aug 2, 2017

@mgravell I was able to repro the issue with a simple console app and pinned the issue down to a custom implementation of IDictionary<Tkey, TValue>. When serialized with 2.1.0, deserializing after upgrading to v 2.3.1 throws the above error.
I will email you the sample console app at marc.gravell@gmail.com

newton3 commented Aug 2, 2017

@mgravell I was able to repro the issue with a simple console app and pinned the issue down to a custom implementation of IDictionary<Tkey, TValue>. When serialized with 2.1.0, deserializing after upgrading to v 2.3.1 throws the above error.
I will email you the sample console app at marc.gravell@gmail.com

@newton3

This comment has been minimized.

Show comment
Hide comment
@newton3

newton3 Aug 2, 2017

having difficulty with email - so created a repo for you to check out the code - https://github.com/newton3/protobuf-2.3.1-error

newton3 commented Aug 2, 2017

having difficulty with email - so created a repo for you to check out the code - https://github.com/newton3/protobuf-2.3.1-error

@mgravell

This comment has been minimized.

Show comment
Hide comment
@mgravell

mgravell Aug 2, 2017

Owner
Owner

mgravell commented Aug 2, 2017

@newton3

This comment has been minimized.

Show comment
Hide comment
@newton3

newton3 Aug 2, 2017

Thanks for your quick response.. that does solve the issue!
Will wait for your complete diagnosis.

newton3 commented Aug 2, 2017

Thanks for your quick response.. that does solve the issue!
Will wait for your complete diagnosis.

@newton3

This comment has been minimized.

Show comment
Hide comment
@newton3

newton3 Aug 3, 2017

@mgravell is there any workaround to apply globally or at the type level? the Dictionary is being used at a number of places and it is a huge change to find and fix all the members. Are you going to open a new issue to track this ?

newton3 commented Aug 3, 2017

@mgravell is there any workaround to apply globally or at the type level? the Dictionary is being used at a number of places and it is a huge change to find and fix all the members. Are you going to open a new issue to track this ?

@mgravell

This comment has been minimized.

Show comment
Hide comment
@mgravell

mgravell Aug 3, 2017

Owner
Owner

mgravell commented Aug 3, 2017

@mgravell

This comment has been minimized.

Show comment
Hide comment
@mgravell

mgravell Aug 4, 2017

Owner
Owner

mgravell commented Aug 4, 2017

@newton3

This comment has been minimized.

Show comment
Hide comment
@newton3

newton3 Aug 4, 2017

yes, thank you!

newton3 commented Aug 4, 2017

yes, thank you!

@mgravell

This comment has been minimized.

Show comment
Hide comment
@mgravell

mgravell Aug 4, 2017

Owner
Owner

mgravell commented Aug 4, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment