Making beqs() useable

[eddelbuettel]

Ok, I started a new branch with

• actually running compileAttributes() so that RcppExports.{cpp,R} is updated
• actually running roxygenize() so that the manual page is created

With that we can actually call beqs(), and I'll commit that shortly. That is the good news.

The bad news is that the returned object is, well, not really up to our standards:

!R> head(res <- beqs("Global Oil Companies YTD Return","GLOBAL"))
   Date Ticker             Short Name         Total Return YTD Market Cap          Price:D-1   P/E         Revenue T12M         EPS T12M
 1 NA   "002506 CH Equity" "GCL SYSTEM INT-A" "1078.202501"    "2513595136.000000" "1.001490"  "2.211897"  "833422051.073852"   "0.408008"
 2 NA   "S92 GR Equity"    "SMA SOLAR TECHNO" "152.222214"     "1339073024.000000" "38.590000" NA          "893484992.000000"   "-4.484000"
 3 NA   "PTG TB Equity"    "PTG ENERGY PCL"   "143.235624"     "590486336.000000"  "0.353585"  "48.333332" "1279731045.178220"  "0.006965"
 4 NA   "ALDW US Equity"   "ALON USA PARTNER" "131.713313"     "1379535744.000000" "22.069027" "7.233236"  "2321284053.504420"  "2.839346"
 5 NA   "SRS IM Equity"    "SARAS SPA"        "127.338135"     "1803096064.000000" "1.896000"  "34.100716" "9271717120.000000"  "0.055600"
 6 NA   "ES FP Equity"     "ESSO S.A.F."      "99.592984"      "819479360.000000"  "63.750000" NA          "14152000000.000000" "-13.029999"
 R> 

Returning everything as character is better than not having the function, but really not show this should be done.

@csrvermaak @wmorgan85 How can I know what the returned fields are going to? Is this is a fixed, or computable format? From the looks the above should be Date, char, char, numeric, int, numeric, numeric, int, numeric -- and I could agree to drop ints for all numeric if we must. But Date and character types are separate. This fits a data.frame to the tee, so let's build a proper one.

[eddelbuettel]

In other words, is the format (seen from running print() on the returned message) assumed to be fixed:

$ r -lRblpapi -e'print(head(beqs("Global Oil Companies YTD Return","GLOBAL","ENGLISH","GENERAL","20150930",verbose=TRUE)))'
Sending Request: BeqsRequest = { screenName = "Global Oil Companies YTD Return" screenType = GLOBAL Group = "GENERAL" languageId = ENGLISH overrides[] = { overrides = { fieldId = "PiTDate" value = "20150930" } } \
}
Processing Response
BeqsResponse = {
    data = {
        fieldDisplayUnits = {
            Ticker = ""
            Short Name = ""
            Total Return YTD = ""
            Market Cap = ""
            Price:D-1 = ""
            P/E = ""
            Revenue T12M = ""
            EPS T12M = ""
        }
        securityData[] = {
            securityData = {
                security = "002506 CH"
                fieldExceptions[] = {
                }
                fieldData = {
                    Ticker = "002506 CH"
                    Short Name = "GCL SYSTEM INT-A"
                    Total Return YTD = 1078.202501
                    Market Cap = 2527281408.000000
                    Price:D-1 = 1.001491
                    P/E = 2.211897
                    Revenue T12M = 833422051.073852
                    EPS T12M = 0.408008
                }
            }
            [.... many more deleted ...]

[csrvermaak]

Agreed, not ideal. The result will always be a mixture of strings and numbers, thus the only usable object in Rcpp (besides building the DF in C++) is the CharacterMatrix. We can convert to DF either in the C++, or in the R wrapper around beqs_impl, and then convert on a per column basis to numberic if possible. Thoughts?

[csrvermaak]

The order of the columns is not fixed, unfortunately, and the string columns can pop up anywhere in the matrix. The order of the result is determined in EQS (and even then, it is sometimes returned with a different ordering) - thus we cannot make any assumptions about the column type beforehand.

[eddelbuettel]

It can be done. In another project I stored it in something like or Boost Any (or maybe
Boost Variant, its been awhile and I do not have that code anymore) and converted each column based on type when constructing the data frame before return.

And of course here @armstrtw also fixed that issue here for bds():

R> res <- bds("GOOG US Equity", "TOP_20_HOLDERS_PUBLIC_FILINGS")
R> head(res, 3)
  Amount Held       Country Filing Date     Holder Name   Institution Type
1    21733002           n/a  2015-09-25 PAGE LAWRENCE E                n/a
2    21202715           n/a  2015-09-30     BRIN SERGEY                n/a
3    16646344 UNITED STATES  2015-06-30       BLACKROCK Investment Advisor
  Latest Change                            Metro Area Percent Outstanding
1        -16670                                   n/a                6.32
2        -16666                                   n/a                6.16
3        155303 New York City/Southern CT/Northern NJ                4.84
  Portfolio Name  Source
1            n/a  Form 4
2            n/a  Form 4
3            n/a ULT-AGG
R> sapply(res, class)
        Amount Held             Country         Filing Date         Holder Name 
          "numeric"         "character"              "Date"         "character" 
   Institution Type       Latest Change          Metro Area Percent Outstanding 
        "character"           "numeric"         "character"           "numeric" 
     Portfolio Name              Source 
        "character"         "character" 
R> 

We wanted to revisit this issue amd code anyway as there may now be better data frame proxies out there. Or maybe not.. In any event, we have not gotten to it yet.

[eddelbuettel]

I have something very crude in a branch. Currently covers only character and double:

~/git/rblpapi$ r -lRblpapi -e'print(head(beqs("Global Oil Companies YTD Return")))'
            Ticker       Short.Name Total.Return.YTD Market.Cap  Price.D.1
1 002506 CH Equity GCL SYSTEM INT-A       1078.20250 2520166912  1.0014905
2    S92 GR Equity SMA SOLAR TECHNO        152.22221 1339073024 38.5900002
3    PTG TB Equity   PTG ENERGY PCL        143.23562  591326208  0.3535846
4   ALDW US Equity ALON USA PARTNER        131.71331 1379535744 22.0690269
5    SRS IM Equity        SARAS SPA        127.33814 1803096064  1.8960000
6     ES FP Equity      ESSO S.A.F.         99.59298  819479360 63.7500000
        P.E Revenue.T12M     EPS.T12M
1  2.211897    833422051   0.40800850
2  0.000000    893484992  -4.48400007
3 48.333332   1279731045   0.00696470
4  7.233236   2321284054   2.83934554
5 34.100716   9271717120   0.05560001
6  0.000000  14152000000 -13.02999878
~/git/rblpapi$

[eddelbuettel]

I did some more cleanup and now committed the branch.

@csrvermaak That is a good test for your emerging git skills. Do a checkout and then see if you can build in the branch :)

[csrvermaak]

I'm all over it like a rash :)