Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Zero-copy binary reading from the reader's buffer #1712

Open
roji opened this issue Nov 15, 2017 · 6 comments
Open

Zero-copy binary reading from the reader's buffer #1712

roji opened this issue Nov 15, 2017 · 6 comments
Assignees
Milestone

Comments

@roji
Copy link
Member

roji commented Nov 15, 2017

With the new Span introduced in C# 7.2, we should support it for PostgreSQL bytea.

@roji roji added this to the 3.3 milestone Nov 15, 2017
@roji roji self-assigned this Nov 15, 2017
@roji
Copy link
Member Author

roji commented Dec 24, 2017

Some notes:

  • Wait for 2.1, because right now there are no APIs for reading/writing Spans (e.g. in Stream)
  • Possibly add an optimization for bytea for reading a ReadOnlySpan which points into the Npgsql buffer, in cases where the entire value fits in the buffer. That's zero-copy bytea reading, pretty cool.
  • Unrelated to bytea - remove the tempCharBuffer used for skipping, allocate on the stack instead.

@roji roji modified the milestones: 4.0, 4.1 May 12, 2018
@roji roji modified the milestones: 4.1, Backlog Aug 26, 2019
@Brar
Copy link
Member

Brar commented Apr 18, 2021

Looking at this again as part of triage with @roji. Some more comments:

  1. Add an NpgsqlDataReader.GetBytes overload just like the existing one, but which accepts a Span instead of byte[]. Think about async though - maybe GetBytesAsync which accepts a Memory. These should be ADO.NET APIs.
  2. Allow NpgsqlParameter to accept a Memory<byte>, which we then write (just as an alternative to byte[]).

Finally, we could optionally:

  1. Add an API which returns a ReadOnlySpan<byte> directly into the internal buffer. The requirements are:
    a. SequentialAccess isn't used (so that we know the entire buffer is in memory)
    b. When the next row is read, the returned ReadOnlySpan becomes invalidated. It's up to the user to code correctly here...

@NinoFloris
Copy link
Member

We have streams filling this need IMO.

Additionally if we want a more direct translation we'll have to wait until ref struct types can be used as generic arguments.

@roji let me know what you want to do with this issue.

@roji
Copy link
Member Author

roji commented Aug 15, 2023

We have streams filling this need IMO.

There's still the more extreme version of providing access to Npgsql's internal buffer (zero copy), with all the lifecycle trouble that would entail. Otherwise I agree that this would basically be sugar over the user getting a stream and calling the methods there; and given that NpgsqlDataReader already has a very loaded public API surface, I'd avoid adding non-essential sugar there.

Maybe keep this open to have the zero-copy variant somewhere on our radar?

@NinoFloris
Copy link
Member

Right, if we'd expose ROSeq as an alternative to Stream we would be able to satisfy that.

@roji
Copy link
Member Author

roji commented Aug 15, 2023

Yep

@roji roji changed the title Support reading/writing Span to/from bytea Zero-copy binary reading from the reader's buffer Aug 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants