Simplify mi query #11
Conversation
temporary XML/CSV files). Note we still do not communicate with BaseX directory via rjava since it is not yet apparent to me that the additional complexity is worth the benefit. We have two ways to communicate: * A "localDBConn" that calls BaseX in standalone mode. * A "remoteDBConn: that calls BaseX HTTP Server via REST api. A user can then call runQueries with: * Either of those two DB connection objects * An MI query (either the XML or a query for the XML) * list of scenario * list of regions And the user gets back the query results as a "long" data.frame Still TODO: * A lot of error checking * Maybe unit tests * Change around the rest of rgcam to use the new capabilities. * Add utilities to parse a Batch Query file and run with this.
method of running batch query with the new one in addScenrio.
(especially with local DB queries). We are now using readr to parse_csv and dplyr to aggregate MI query results.
returning the data in runQuery.
using the utility shQuote.
performance reasons.
Begin to consolidate functionality so that addQueryTable is the backbone for adding data to the project and is called by drivers such as: * Running a batch query file (addScenario) * Running a single query (newly added addSingleQuery) * Parsing batch query CSV results (newly added addMIBatchCSV) Note, each of the new methods for adding data uses the same paradigm as addScenario.
that this version of rgcam relies on heavily. WARNING: this version is using an updated version of BaseX which is not compatible with Java 1.6.
and description. Set username/password for remoteDBConn using the httr::authenticate instead of in the URL to avoid odd characters in the URL problems.
Fix some bugs in handling empty scenario list.
in the DB results (such as land queries) would sometimes report the scenario name/date multiple times for the same row of data. Typo and doc fixes. Add parallel package as a Suggests.
the ModelInterface directly (since `pipe` doesn't support it) and move the flag to the localDBConn. Fix up tests so they now run with the simplify-mi-query changes.
the issues trying to maintain this attribute. Tests have been updated. Fix bug that 'date' attr wasn't being added in `addQueryTable` that was causing a test to fail.
…ueries in the vignette examples. Error: processing vignette 'rgcam-usecases.Rmd' failed with diagnostics: 4 simultaneous processes spawned Execution halted
Update some documentation, warnings for consistency.
|
Oops, I should bump the version number.. but to what? |
|
Let's update to 0.4.0 for now. We're getting close to being able to go to 1.0, but I'd like to leave us some flexibility to rejigger the interface once we get some feedback from live users. Once we've had a chance to incorporate their feedback, we can lock that in. |
|
Hopefully I'll be able to review and merge this week. I've got a couple of other things ahead in the queue. |
Codecov Report
@@ Coverage Diff @@
## master #11 +/- ##
==========================================
+ Coverage 85.41% 92.13% +6.71%
==========================================
Files 5 5
Lines 384 445 +61
==========================================
+ Hits 328 410 +82
+ Misses 56 35 -21
Continue to review full report at Codecov.
|
Codecov Report
@@ Coverage Diff @@
## master #11 +/- ##
==========================================
+ Coverage 85.41% 92.13% +6.71%
==========================================
Files 5 5
Lines 384 445 +61
==========================================
+ Hits 328 410 +82
+ Misses 56 35 -21
|
|
I've been slacking on this one. I'll get to it today or tomorrow. |
|
@pralitp Is it necessary to close a db connection opened with Also, when I test the package I get a system alert indicating that R is asking for a firewall exception. I'm guessing it's related to opening up a remote connection, for testing. Is this expected? Is there any way we can avoid this sort of alert? When you run a shiny app it starts a server on the loopback interface, and it doesn't trigger an alert, so it ought to be possible. |
No, it is not necessary to close them. At create these objects only track the information they need to run queries on a database. Any sort of connection is open and closed each time a query is run (
No it is not expected and I don't recall seeing that. I have not written any tests for querying a remote DB server since I didn't know how to configure and launch such a server reliably from the tests (would travis allow such a thing?). There is one exception where I have a test to ensure failure when opening a |
|
I'll investigate further and see what I can come up with. If you deny the firewall request when it comes up it doesn't recur, and there doesn't seem to be any impact on the package functionality. So, I'm pretty sure it will be an easy fix once we locate the source of the problem. |
The filename is converted to a connection using localDBConn().
|
Made a few documentation updates and whatnot. I'll take a quick stab at tracking down the firewall alert tomorrow, but if it's too much of a pain to locate, we'll just live with it for now. |
|
It's definitely the |
A whole suite of changes. Originally intending to make querying the model interface easier by no longer having to generate temporary batch file templates writing CSV to disk and reading it back in. But feature creep quickly set in as I starting using it to do analysis.
Now queries are sent to the model interface and queries results parsed without going to disk. We did not go as far as using the rJava package mostly because I wanted to add a feature where queries can be made on a remote server in which queries are made and results transferred over HTTP requests. If a user is solely running queries on a remote sever then they do not even need to have Java installed.
Thus
addScenarionow takes as a first argument an S3 generic object which can be either alocalDBConnor aremoteDBConnwhich behind the scenes runs the queries appropriately.In addition some more "add" methods to import data into a project. The first is
addSingleQuerywhich is similar toaddScenarioexcept a user doesn't have to bother writing a full batch query file it the only need to make a couple of queries. AlsoaddMIBatchCSVwhich is mostly leveraging functionality that was already there (parse a Model Interface batch CSV results file) to add to a project file but provides a consistent interface, for instance to specify transformations. To do this cleanly I refactored some code so each of these methods are just usingaddQueryTableunder the hood. This also resolves #7.There are a number of other methods that are used under the hood but are exposed for flexibility. They are all explained better than they could be here in the "rgcam-usecases" vignette.