Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Problem with PostgreSQL byte[] serialization/deserialization #191

Merged
merged 3 commits into from Mar 10, 2013

Conversation

Projects
None yet
4 participants
Contributor

AkosLukacs commented Mar 5, 2013

In PostgreSQLDialectProvider, the insert command actually inserts a base64 encoded value instead of postgres-escaped valid "bytea" representation of the data.
For reading back the data PostgreSQLDialectProvider.ConvertDbValue falls back to OrmLiteDialectProviderBase.ConvertDbValue for byte[], and that uses TypeSerializer.DeserializeFromStream<byte[]>. That is actually invalid, but it seems that it works with base64 encoded strings, as OrmLite's Insert does.

The InsertParam<T> method does correctly insert values to the db, that causes any Select<T> reading that value to fail.
First commit contains a couple of failing test in various combination of manual insert and select, Insert<T>, InsertParam<T>, Select<T>.

Second commit changes serialization (copied from Npgsql2's code: https://github.com/franciscojunior/Npgsql2/blob/master/src/NpgsqlTypes/NpgsqlTypeConverters.cs ), and deserialization for byte[] (just returns it).

Note:

This could be a breaking change for PostgreSQL users, if they used Insert<T> and Select<T> after inserting byte[]s in Postgres actually worked. But not really likely, since commit 584765e was publicly released in 3.9.38. two days ago).

@brainless83 - please take a look at this, if you use Postgres.

Owner

mythz commented Mar 5, 2013

Also adding @tomaszkubacki who also maintains the Postgres driver.

Member

tgrassau commented Mar 6, 2013

@AkosLukacs you are right. the default behavior that is implemented in OrmLiteDialectProviderBase when writing byte[] values, is to base64-encode them and store them as string values in the database. when reading the value from the db the value is converted back to byte[]. This last step was missing pre 584765e .

Of course it would be better to store the actual byte[] without converting it to a base64 encoded string. For the SQL Server and MySql dialect this is already implemented.

As I mostly use SQL Server and MySql I think it would be great if @tomaszkubacki could look into that

Member

tomaszkubacki commented Mar 6, 2013

W dniu 06.03.2013 15:30, Thomas Grassauer pisze:

@AkosLukacs https://github.com/AkosLukacs you are right. the default
behavior that is implemented in OrmLiteDialectProviderBase when writing
byte[] values, is to base64-encode them and store them as string values
in the database. when reading the value from the db the value is
converted back to byte[]. This last step was missing pre 584765e
584765e .

Of course it would be better to store the actual byte[] without
converting it to a base64 encoded string. For the SQL Server and MySql
dialect this is already implemented.

As my mostly use SQL Server and MySql I think it would be great if
@tomaszkubacki https://github.com/tomaszkubacki could look into that


Reply to this email directly or view it on GitHub
#191 (comment).

will take a look till end of week - a bit busy now

@tomaszkubacki tomaszkubacki was assigned Mar 9, 2013

@tomaszkubacki tomaszkubacki added a commit that referenced this pull request Mar 10, 2013

@tomaszkubacki tomaszkubacki Merge pull request #191 from AkosLukacs/postgres_bytea
Problem with PostgreSQL byte[] serialization/deserialization
2d0adb6

@tomaszkubacki tomaszkubacki merged commit 2d0adb6 into ServiceStack:master Mar 10, 2013

Member

tomaszkubacki commented Mar 10, 2013

ok so this commit converts blobs to bytea Escape Format http://www.postgresql.org/docs/9.1/static/datatype-binary.html#AEN5231. (@AkosLukacs would be interesting to is if using bit shifting + StringBuffer.Append in ToBinary, while converting base10 to base8 is really faster than using Convert.ToString(byte,8)). Anyway good job :)

Member

tomaszkubacki commented Mar 10, 2013

@mythz and forgot to add: this is indeed a breaking change and need mention in release notes that we NO longer store byte[] as base64 converted strings in PostgreSQL provider

Contributor

AkosLukacs commented Mar 10, 2013

Thanks.
Don't know about performance, but if someone finds a faster way, tell it to the Npgsql2 devs, since the escaping itself is copied from there.
Actually 3.9.38 was a breaking change for me, since previously I used a manual parameterized insert, and that Select<T> read the byte[] back correctly. Probably it changed around 584765e but didn't notice, since didn't pull the latest source in a couple of weeks.

Owner

mythz commented Mar 11, 2013

@tomaszkubacki cool, do you just want to dropt a note on the ServiceStack G+ Group:
https://plus.google.com/u/0/communities/112445368900682590445

I'm trying to make that the canonical source for all news-worthy items and announcements for all ServiceStack libraries

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment