AVRO-2014 Add support for custom streams for the DataFile interfaces.

[mariusvniekerk]

Downstream consumers will still have to implement their own relevant
Input/Outputstream subclasses.

[mariusvniekerk]

Rationale for this change is that we want to add avro support to apache arrow cpp and we need to easily be able to interact with non filesystem datafiles.

[mariusvniekerk]

cc @vimota @thiru-apache

[vimota]

LGTM, perhaps add a unit test in DataFileTests.cc that covers this new interface?

[thiru-mg]

@mariusvniekerk Looks good to me. As @vimota mentioned, a unit test would be useful. Please assign the ticket to yourself in JIRA: https://issues.apache.org/jira/browse/AVRO-2014. Thank you

[mariusvniekerk]

@thiru-apache don't think my user on apache jira has enough power oddly

[thiru-mg]

@mariusvniekerk What is your Jira id? I'll assign the ticket to you.

[steinio]

Hi.
I am currently trying to create a DataFileWriter with an OutputStream as input argument, instead of a const char* filename, as I want to send the DataFileWriter encoded data over a Kafka buffer.

There is no examples on how to do this, so my attempt was to do this:

c::scheme scheme;
scheme.x = 1;
scheme.y = -1

std::unique_ptr<avro::OutputStream> out = avro::memoryOutputStream();
avro::DataFileWriter<c::scheme> dfw(std::move(out), scheme);

which compiles, but when I run it I get an error due to
filename_(NULL),
in the DataFileWriterBase constructor.
Have I done anything wrong when setting up the DataFileWriter, if so, how do I go about setting this up?
If not, what can this issue come from?

I noticed that you were going to make a test for this in DataFileTests.cc, but can't find any test for this OutputStream support in DataFileWriter.

[ghost]

Hi - we are working to upgrade our codebase from 1.8 to 1.9, and we noticed that we also modified DataFileWriter to accept an OutputStream. However, our API takes the OutputStream by reference, instead of unique_ptr. Apart from being more consistent with, e.g., Encoder::init(OutputStream& os), this is also a bit more useful. See the comment above: how do you get back the stream that was written to out if out is moved into dfw?

Can I suggest to at least add an alternative API to take the stream by ref, and possibly deprecate this one? I am willing to open a PR.

[thiru-mg]

Hi, thanks for your interest.

Please go ahead. The current implementation uses a unique_ptr and hence owns the OutputStream. If a reference is passed, this ownership model will change. We should somehow indicate the the OutputStreamBase class that it does not own the underlying ostream.