Error in bdp_Impl: attempt to set an attribute on NULL

[RockScience]

This works

con = Rblpapi::blpConnect()
Rblpapi::bdp(securities = "DE US Equity", fields = "CRNCY", con = con)
Rblpapi::blpDisconnect(con = con)

but this returns an error:

con = Rblpapi::blpConnect()
Rblpapi::bdp(securities = "IP Index", fields = "CRNCY", con = con)
Rblpapi::blpDisconnect(con = con)

Error in bdp_Impl(con, securities, fields, options, overrides, identity) : 
   attempt to set an attribute on NULL

Maybe it is stupid to look for the CRNCY of IP Index, but I would have expected to get a NULL or an empty data.frame, not an error.

Thank you

packageVersion("Rblpapi")
[1] '0.2.2.6.2.3'


R.version                       
platform       i386-w64-mingw32            
arch           i386                        
os             mingw32                     
system         i386, mingw32               
status                                     
major          3                           
minor          2.0                         
year           2015                        
month          04                          
day            16                          
svn rev        68180                       
language       R                           
version.string R version 3.2.0 (2015-04-16)
nickname       Full of Ingredients

[RockScience]

note that this works

Rblpapi::bdp(securities = c("IP Index","DE US Equity"), fields = "CRNCY", con = con)

(I paste here the dput of the result)

structure(list(CRNCY = c("", "USD")), class = "data.frame", .Names = "CRNCY", row.names = c("IP Index", "DE US Equity"))

[eddelbuettel]

How do you suggest we deal with that programmatically?

We only respond to what we get from Bloomberg and if the data is bunk, there is little we can do. We cannot know before the query whether it will fail or not i.e. guess whether a field is appropriate for the security in question or not. This seems like user error to me. So for an unanswerable query you get an empty result, which is currently an error condition.

If you want to make that nicer, go for it. We generally take well written pull requests.

(And you don't need to pass con, or call blpDisconnect(con). The latter is empty.)

(Also don't use quote > when you mean an R block. Hit the edit button above to see what I set.)

[armstrtw]

Yes, it's related to lazy initialization. The response vector for 'CRNCY' is never initialized because there is no data from bbg.

We should revisit that part of the code.

[eddelbuettel]

With that you get the 'assigned to Whit' label.

[RockScience]

Thank you for your feedbacks and for considering my suggestion. I have indeed looked at the source code of bdp, However I am less fluent in c++ than in R and couldnt find the exact replacement code. From the error message it seems that we are assigning an attribute on a NULL object.
I agree we cannot know in advance what will be the size of the object returned by the API. But once we have got it, could we test for its size before manipulating it?

[armstrtw]

haha. nice. I'll try to knock this out.

I think we can bring back the field search. just may have to reassign the Date/time vector to a Date vector later. but I'm much more comfortable w/ Rcpp now, so perhaps that will be easier...

[eddelbuettel]

(Re DateVector and DatetimeVector in Rcpp: Those are bound to change "one day" as they are terrible. See eg here in Rblpapi itself for a (so-far ill-named) POSIXtVector alternative.)

[armstrtw]

That should be fine. If memory serves, the issue was that bbg would frequently return 'date-time' for date fields or possibly 'date' for date-time fields. However, w/ lazyframe or whatever container we use. When we see the first instance of the field, we can simply reassign it to the correct type and delete the old vector. I guess I thought this was too inelegant the first time around...

[pgarnry]

I also came across this issue today. Coming back to Dirk's initial question How do you suggest we deal with that programmatically?

I sort of lean towards not doing anything on the Rblpapi side because the bdp function just responds to the output from Bloomberg and the end user can easily just catch the error and handle it (see my example below). The error arises only if there is no data so why not just catch the error with tryCatch and return NA?

If handling the issue makes the bdp function slower because of extra checks I would prefer to leave the bdp function as it is and urge end user to solve the issue on their side.

Do you guys agree?

fields <- c("BEST_EPS", "BEST_EPS_YOY_GTH", "BEST_EPS_6MO_PCT_CHG",
"BEST_SALES", "BEST_SALES_YOY_GTH")

tryCatch({Rblpapi::bdp(securities = "7912 JT Equity",
                       fields = fields,
                       overrides = structure("1FY",
                                             names = "BEST_FPERIOD_OVERRIDE"))},
         error = function(e) {rep(NA, length(fields))})

# returns NA with 

[eddelbuettel]

The idea is right but the handling has to happen at the C++ level as @armstrtw has hinted at, and volunteered to do, eventually...

[armstrtw]

It's coming. As usual, it's taking a little longer given what's been happening in the markets.

[pgarnry]

Thank you both, it will a valuable commit when implemented at the C++ level

@armstrtw makes sense. No rush since users can handle it for now outside the bdp function

[armstrtw]

you can try the current master if you want. should be working. but limited testing has been done so far.

[pgarnry]

I just installed the current master and my example still produces the same error. Do you also get the error if you run my example (without tryCatch)?

[armstrtw]

bdp(securities = c("DE US Equity","IP Index"), fields = "CRNCY")
CRNCY
DE US Equity USD
IP Index

or were you talking about the other example?

[pgarnry]

This example...

library(Rblpapi)

blpConnect()

fields <- c("BEST_EPS", "BEST_EPS_YOY_GTH", "BEST_EPS_6MO_PCT_CHG",
            "BEST_SALES", "BEST_SALES_YOY_GTH")

bdp(securities = "7912 JT Equity", fields = fields,
    overrides = structure("1FY",
                          names = "BEST_FPERIOD_OVERRIDE"))

[armstrtw]

yes. I get all NA for that.

with overrides and without.

[pgarnry]

Thanks. I'll go back and check whether an error occured when I installed the current master.

[armstrtw]

and?

[pgarnry]

An error in the installation of master had duck under the rader. After I found out, I tried every method of installing master

1. devtools::install_github
2. install directly from zip
3. clone from GitHub and trying installing from local repo
   Nevertheless all methods failed and I eventually had to move on with other issues. I am pretty sure it has something to do with being on a Windows environment with no admin rights due to security - possibly the same issue flagged in Windows support #40
   I wanted to post the installation error message here, but then I decided that it is properly better on SO in order to not contaminate this thread.

[eddelbuettel]

1. Check out the source.
2. Edit DESCRIPTION locally to make it your email address in Maintainer:.
3. Run R CMD build.
4. Submit the tarball to win-builder.
5. A couple minutes later, use the emailed response to recover the binary built for you.

[pgarnry]

@eddelbuettel thanks! I'll try your solution first thing tomorrow morning (European time)...

[pgarnry]

R CMD build fails at curl. Trying to get Cygwin, but there is a firewall issue installing it. I'll get it fixed later today. As soon as I have installed current master I will check the "Error in bdp_Impl" throughout my various R scripts and see if the error is eliminated.

[eddelbuettel]

So do that part (ie R CMD build and upload to win-builder) at home and install the resulting binary at work the next morning.

[pgarnry]

Will do. Cannot wait for IT security to move on this...

[pgarnry]

@eddelbuettel it worked. Thanks for your guidance. I learned a great deal along the way.

@armstrtw your C++ implementation works as intended in my various scripts after I have removed the tryCatch functions. A quick question, maybe a bad one because I haven't paid enough attention to the recent commits, but is the output order of fields now identical to the input order? It seems so when I look through the results of my code.

[eddelbuettel]

@pgarnry good to know -- we'll look into releasing 'shorty', in the meantime you now also now how to help yourself to newer binaries.

[armstrtw]

@pgarnry can you close this out? this works now (github master).

[eddelbuettel]

Or @RockScience who originally opened this.

[pgarnry]

This one goes to @RockScience

[RockScience]

@armstrtw thank you for looking into this.
@pgarnry could you share your windows binary of the latest version? (unfortunately I have also only Windows access to the BBG API)
Thank you

[pgarnry]

@RockScience I deleted my local binary after it was build with win-builder. I can send you the build version (0.3.2.2) if you want? If you insist on the binary I might be able to do it this evening (CET) - no promise though

[eddelbuettel]

@RockScience Please scroll up two screens to this message where I explain how to roll your own via win-builder. It really is not that hard.

We will release a new version at some point too.

[RockScience]

Thanks, I managed to create the bin with the win-builder. It passes also my unit tests. This closes the thread. Thank you @armstrtw .

[RockScience]

I reopen temporarly as unfortunately it seems that the following call now fails in the most recent version (I checked and in the CRAN version it works):

con <- Rblpapi::blpConnect()
Rblpapi::bdh(securities="DOENUSCH Index", 
                                         fields="ECO_RELEASE_DT", 
                                         start.date = as.Date('2015-09-26'), 
                                         end.date = as.Date('2016-02-29'), 
                                         include.non.trading.days = FALSE, 
                                         options=NULL, 
                                         overrides=NULL, 
                                         identity=NULL, 
                                         con=con)

# Error: Attempt to access value of element 'ECO_RELEASE_DT'(type: 'Float64') as 'Datetime' type.

Is it the same on your side?

[eddelbuettel]

No, it also bombs for me with the most recent code base:

R> Rblpapi::bdh(securities="DOENUSCH Index", fields="ECO_RELEASE_DT", start.date = as.Date('2015-09-26'), end.date = as.Date('2016-02-29'))
Error: Attempt to access value of element 'ECO_RELEASE_DT'(type: 'Float64') as 'Datetime' type.
R> 

@armstrtw Can you take a look? Bbg appears to have a screw loose with this fieldtype:

R> Rblpapi::bdh(securities="DOENUSCH Index", fields="ECO_RELEASE_DT", start.date = as.Date('2016-02-01'), end.date = as.Date('2016-02-29'), include.non.trading.days = FALSE, verbose=TRUE)
HistoricalDataResponse = {
    securityData = {
        security = "DOENUSCH Index"
        eidData[] = {
        }
        sequenceNumber = 0
        fieldExceptions[] = {
        }
        fieldData[] = {
            fieldData = {
                date = 2016-02-05
                ECO_RELEASE_DT = 20160211.000000
            }
            fieldData = {
                date = 2016-02-12
                ECO_RELEASE_DT = 20160218.000000
            }
            fieldData = {
                date = 2016-02-19
                ECO_RELEASE_DT = 20160225.000000
            }
            fieldData = {
                date = 2016-02-26
                ECO_RELEASE_DT = 20160303.000000
            }
        }
    }
}
Error: Attempt to access value of element 'ECO_RELEASE_DT'(type: 'Float64') as 'Datetime' type.
R> 
R> fieldInfo("ECO_RELEASE_DT")
                  id       mnemonic datatype ftype
ECO_RELEASE_DT DX252 ECO_RELEASE_DT Datetime  Date
R> 

[armstrtw]

I think I would classify this as a bbg fail...

This call works as a bdp call:

> bdp(c("CPI Index","DOENUSCH Index"),"ECO_RELEASE_DT")
               ECO_RELEASE_DT
CPI Index          2016-03-16
DOENUSCH Index     2016-03-03

and the return type is 'Date' as it should be (and as fieldInfo says it is).

but fails as BDH, as bbg decides to send the data as a double, in YYYYMMDD.HHMMSS format... I suppose.

I'm really not sure what to do here. If we can't rely on fieldInfo, then we would have to build our own field->type map which is not something I think is sustainable. Especially when it changes from bdp to bdh, we would need to build a separate map for each call.

@wmorgan85 can you please offer some advice?

[RockScience]

@armstrtw indeed this field is both a reference and a historical field. Maybe we need an argument in fieldInfo to specify the category of data?

[armstrtw]

you can look:
http://www.bloomberglabs.com/api/content/uploads/sites/2/2014/07/blpapi-developers-guide-2.54.pdf

but it seems bbg does not distinguish between the two services in terms of fieldInfo.

[armstrtw]

I was thinking about this last night, and perhaps the more pragmatic approach is just to test for dates returned as doubles in the populateDfRow function and convert them:
https://github.com/Rblp/Rblpapi/blob/master/src/blpapi_utils.cpp#L216

My thinking is that for the other types (bool, double, int), the bbg functions getValueAsXXX will do an automatic conversion if the type is convertible. So, even if bbg lies to us about the actual types being returned, let's say getFieldType tells us something is an Int32, and bbg actually returns a double. The getValueAsInt32 function should be able to convert it without complaining (at least that is my working assumption for now).

If we continue to see exceptions (if the getValueAsXXX functions are not able to autoconvert), then we will have to try another approach.

[eddelbuettel]

👍 -- sounds good

[armstrtw]

the patch works.

will add a test case as well.

library(Rblpapi)
bdh("DOENUSCH Index","ECO_RELEASE_DT",start.date=as.Date('2016-02-01'),end.date=as.Date('2016-02-29'),verbose=TRUE)HistoricalDataResponse = {
securityData = {
security = "DOENUSCH Index"
eidData[] = {
}
sequenceNumber = 0
fieldExceptions[] = {
}
fieldData[] = {
fieldData = {
date = 2016-02-05
ECO_RELEASE_DT = 20160211.000000
}
fieldData = {
date = 2016-02-12
ECO_RELEASE_DT = 20160218.000000
}
fieldData = {
date = 2016-02-19
ECO_RELEASE_DT = 20160225.000000
}
fieldData = {
date = 2016-02-26
ECO_RELEASE_DT = 20160303.000000
}
}
}
}
date ECO_RELEASE_DT
1 2016-02-05 2016-02-11
2 2016-02-12 2016-02-18
3 2016-02-19 2016-02-25
4 2016-02-26 2016-03-03

[RockScience]

Works on all my examples so far. Thank you.