[CALCITE-3560] Additional calcite.util.Source implementation for generic text source (eg. CharSource)#1632
Conversation
example/csv/src/main/java/org/apache/calcite/adapter/csv/JsonEnumerator.java
Outdated
Show resolved
Hide resolved
…ric text source (eg. CharSource) Currently calcite calcite.util.Source interface can be built only from File or URL. This forces data to be stored on disk or accessed remotely using URL api. This proposal adds another Source implementation on the top of CharSource so calcite adapters can operate on in-memory elements like String or generic text sources.
asereda-gs
force-pushed
the
source-reader
branch
from
e83c7a4 to
a55bd51
Compare
| @Override public Source relative(final Source source) { | ||
| throw unsupported(); |
There was a problem hiding this comment.
It is used in
So it looks like the implementation should be not just a CharSource, but it should be like a virtual file system, so it is not clear how you want CharSource to blend into csv and other adapters.
There was a problem hiding this comment.
CsvSchema won't create Sources on the top of CharSource. New (CharSource) Source implementation was created for users who would instantiate JsonTable directly (via constructor).
On your point with virtual file system, I think Source API is a little "coarse" in the sense that it exposes two functionalities:
- Operations on FS / URL path elements (trim / append / relative)
- supplier of
InputStream/Reader(a laCharSource/ByteSource)
Obviously you can't do (1) when your data-source is String and not a File.
I was thinking of introducing something like Locator API where one can compose / operate on path elements and then expose CharSource from Locator. To some extent Source = Locator + CharSource. This requires more thought and I'm still not convinced that it worths it.
There was a problem hiding this comment.
@vlsi are you OK if I merge this PR as is ?
I don't think there is a nice work-around for Source API without "origin" (file/url)
| * @return {@code Source} delegate for {@code CharSource} (can't be null) | ||
| * @throws NullPointerException when {@code source} is null | ||
| */ | ||
| public static Source fromCharSource(CharSource source) { |
There was a problem hiding this comment.
Is this significantly better than java.io.Reader and/or java.io.InputStream?
There was a problem hiding this comment.
Yes. The idea is that Reader / InputStream are "single-pass" in the sense that you can't re-read them. CharSource is a supplier of Reader similarly to File / URL.
You can't provide InputStream to Source because it can be read only once.
There was a problem hiding this comment.
Thanks, I see. That is fine then.
2 participants
Currently calcite
calcite.util.Sourceinterface can be built only from File or URL. This forces data to be stored on disk or accessed remotely using URL api.This proposal adds another Source implementation on the top of CharSource so calcite adapters can operate on in-memory elements like String or generic text sources.