New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot restrict DataFrame to certain mapping field #497
Comments
Thanks for the snippet; I'm currently on PTO however once I'll try it out as soon as I'm back. |
@analyticswarescott Unfortunately I'm unable to reproduce the problem. What version of Es-hadoop and Spark are you using? Are you by any chance on Spark 1.3?
Can you post your entire log and potentially turn on logging on the rest package ( Thanks |
Your example test query string does not specify a set of fields to return, I will provide more information as soon as I am able. --Scott On Tue, Jul 28, 2015 at 11:49 AM, Costin Leau notifications@github.com
|
I see. That's a bug - fields should not be used with a DataFrame - it's the DataFrame itself that specifies the fields On 8/4/15 8:27 PM, Scott wrote:
Costin |
So, to clarify, the .esDF method is only designed to return a DataFrame --Scott On Tue, Aug 4, 2015 at 7:41 PM, Costin Leau notifications@github.com
|
Good point - this feature is in there but it is not properly exposed. One could create an RDD based on a query with just the needed fields and then associate a schema with it but it's overkill. Cheers, |
Thanks. I can use the RDD approach for now, and will stay tuned. --Scott On Tue, Aug 4, 2015 at 10:48 PM, Costin Leau notifications@github.com
|
This effectively allows the user to specify a custom schema that cherry-picks the fields inside a mapping instead of using all of them relates #497
This feature is in master and 2.x; see the docs here, in particular the "Controlling the DataFrame schema". You can try it out through the dev builds. Cheers, |
Thanks very much for keeping me up to date. We'll be checking it out soon! --Scott On Thu, Sep 10, 2015 at 11:35 PM, Costin Leau notifications@github.com
|
@analyticswarescott Any update? Did you manage to try it out? Wanted to know whether the current feature is properly designed (and rich enough). |
Closing the issue. |
The DataFrame returned by JavaEsSparkSQL.esDF contains Scala Buffers when a query string is specified, but not when the simpler overload is used. Code snippet and log output is below.
Log output from this snippet:
The text was updated successfully, but these errors were encountered: