-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to specify or change configuration #13
Comments
Currently the loading strategy is not configurable. We've prepared for it to be configurable, that's why it's in The same goes for the polling strategy, which could be useful to be able to configure, but also useful to be able to replace with your own implementation in some cases. What's your use case? Do you want to switch to using |
First it is unclear to me still what the 'current' loadings strategy is -- the docs and the implementation seem misaligned. when I went to look I discovered its not changeable -- `By setting a client request token on a query execution you can make Athena reuse a previous result set if the exact same query has already been run. If you run the same query multiple times this can save money and improve performance. ` I was rerunnign the same query and specifying the query token but it still took as long. So .. on the path to see if I could implement that -- I thought maybe if I could get the S3 URL to the results then I could fetch it myself -- and yes I can -- but its not so useful without the metadata -- and I dont want to parse that when you have a parser already -- So back to step one -- there is no obvious way to cache query results without making another copy first -- I was hoping to deduce a way to reuse the polling or results code via the various configurable features but found that in fact they are not configurable. So to answer your question -- my use case is neither -- although maybe it might be both if they were configuable , hard to tell. |
If there is a misalignment, it would be great to fix it, but as far as I can see, the documentation specifies that the default loading strategy is to load from S3, bypassing the Athena API, and the implementation selects the
The feature is indeed intended and described as to be used to ensure exactly-once processing, but we are actively using it for the caching benefit described in the documentation. Arguably, the README could be a bit clearer on what the original purpose of the token is, and that it can also be used for caching. We could probably also make it more clear that you must provide the same token every time you execute the query, including the first one. The way it is written now, it sounds as if you could provide a token to gain access to a previously executed query, which is not true. |
Here is a specific use case for overloading configuration:
The way to make it sort of work work with 4.0:
The default Athena driver, unfortunately, is auto-registering itself with default configuration upon class-load and therefore leaves no opportunity to inject a custom configuration. Original non-open-source Athena driver sort of dealt with this problem by having a configuration parameter that is a fully-qualified class name that would be doing configuration work. I'd argue this is pretty nasty and not a good way to do these things. There are many ways of dealing with configuration injection here, but none of them are decent. I'd say half-bad solution would be to have a base non-self-registering driver. LMK if you want an MR for this. |
I cannot find how to change the or specify the configuration, specifically the loading strategy.
All the members are private and only a few mutators are exposed.
The only place I can see to override is in the Connection() constructor -- but my existing app uses libraries that call the generic JDBC connect().
Would be useful to either expose all the configuration properties or to allow a global override of the factory.
The text was updated successfully, but these errors were encountered: