NullPointerException in getFullOrderBook() in Android 2.2

[veken0m]

This issue happens in Android 2.2 (my phone is running 4.2.1 and is not affected). MtGox and CampBX don't seem to be affected but it might just be matter of running more tests.

I catch the exception and display an error message to enable the user to try again so this isn't a problem for me anymore. It seems to always work on the second try and does not happen very often. Here are the stack traces if anyone wants to look into it:

Bitstamp:
01-20 18:28:24.303: W/System.err(333): java.lang.NullPointerException
01-20 18:28:24.303: W/System.err(333): at com.xeiam.xchange.bitstamp.BitstampAdapters.adaptOrders(BitstampAdapters.java:85)
01-20 18:28:24.303: W/System.err(333): at com.xeiam.xchange.bitstamp.service.marketdata.polling.BitstampPollingMarketDataService.getFullOrderBook(BitstampPollingMarketDataService.java:92)

BTC-E:
01-20 18:17:43.353: W/System.err(312): java.lang.NullPointerException
01-20 18:17:43.363: W/System.err(312): at com.xeiam.xchange.btce.service.marketdata.polling.BTCEPollingMarketDataService.getFullOrderBook(BTCEPollingMarketDataService.java:88)

VirtEx:
01-20 19:36:02.204: W/System.err(782): java.lang.NullPointerException
01-20 19:36:02.204: W/System.err(782): at com.xeiam.xchange.virtex.service.marketdata.polling.VirtExPollingMarketDataService.getFullOrderBook(VirtExPollingMarketDataService.java:94)
01-20 19:36:02.212: W/System.err(782): at com.veken0m.bitcoinium.OrderbookActivity.getOrderBook(OrderbookActivity.java:141)

[mmazi]

Can you post a log with debug level logging for com.xeiam.xchange?
<logger name="com.xeiam.xchange" level="debug" />

Also (this may be a separate issue), a friend got an error with Bitcoinium (I built it for him from source) on HTC Wildfire S (Android 2.3). I don't know if it's the same. But I think the error occured "always", not rarely as in your case. I don't have a log though. Does it work for you on Android 2.3?

[timmolter]

It looks to me like the exchange-specific DTO coming from our REST proxy is null for some reason. This problem reminds me of the failed integration tests that Matija saw, but I did not. Matija, can you reproduce the failed tests still?

[mmazi]

The failed test I saw and Tim didn't were different: certificate issues with CaVirtEx (still always happens) and a signature issue with BTCE (happens occasionally).

[veken0m]

I'm not sure how to obtain a log for com.xeiam.xchange in android...

I'll create an Android 2.3 AVD and I'll look into your friend's issue, Matija. Does he get the issue if he uses the version on Google Play?

[mmazi]

No, I forgot to mention: the version from Google Play works for him.

On Mon, Jan 21, 2013 at 3:53 PM, veken0m notifications@github.com wrote:

I'm not sure how to obtain a log for com.xeiam.xchange in android...

I'll create an Android 2.3 AVD and I'll look into your friend's issue,
Matija. Does he get the issue if he uses the version on the Google Play?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-12500916.

[veken0m]

Just tested Android 2.3.1 (API Level 9) and Android 2.3.3 (API Level 10) and can't seem to reproduce the issue. Perhaps the build you sent him contained unfinished code? I've been reworking some things in Orderbook/Graphing this past week.

[mmazi]

Thanks for testing! I sent him a version a week ago. It could have been
unfinished, or it could have been my mistake (I know nothing about Android
development, maybe I built it incorrectly, but it worked on my Android 4).
If you can't reproduce it, don't worry about it. I'll ask him to test again
when you release a new version.

On Mon, Jan 21, 2013 at 6:17 PM, veken0m notifications@github.com wrote:

Just tested Android 2.3.1 (API Level 9) and Android 2.3.3 (API Level 10)
and can't seem to reproduce the issue. Perhaps the build you sent him
contained unfinished code? I've been reworking some things in
Orderbook/Graphing this past week.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-12507575

.

[timmolter]

I don't know how to see the logs on an Android but try replacing the slf4j jar that we use in XChange with the one designed for Android: http://www.slf4j.org/android/

This may help as well: http://stackoverflow.com/questions/4889782/android-cant-get-logs-with-slf4j-android

One way or another you should be able to see logs coming out at different log levels. Use DEBUG to see everything.

[veken0m]

Thanks Tim! I swapped out the slf4j with the proper android one and now get com.xeiam.xchange stuff in my log.

What's weird is that the NullPointerException happens after I "successfully" obtain the orderbook and about to print it. I say "successfully" because the orderbook fetching is done in a try/catch brace and there is no exception throw then. I need to catch it at a higher level...

Anyways, here's the new log (I trimmed out most of the android stuff):

01-21 14:17:04.689: D/c_.x_.x_.ExchangeFacto_(332): Creating default exchange from class name
01-21 14:17:04.719: W/dalvikvm(332): VFY: unable to find class referenced in signature (Lcom/xeiam/xchange/rest/AllParams;)
01-21 14:17:04.759: I/dalvikvm(332): Could not find method com.xeiam.xchange.rest.AllParams.getPostBody, referenced from method com.xeiam.xchange.btce.service.BTCEHmacPostBodyDigest.digestParams
01-21 14:17:04.759: W/dalvikvm(332): VFY: unable to resolve virtual method 7923: Lcom/xeiam/xchange/rest/AllParams;.getPostBody ()Ljava/lang/String;
01-21 14:17:04.759: D/dalvikvm(332): VFY: replacing opcode 0x6e at 0x0000
01-21 14:17:04.759: D/dalvikvm(332): VFY: dead code 0x0003-0027 in Lcom/xeiam/xchange/btce/service/BTCEHmacPostBodyDigest;.digestParams (Lcom/xeiam/xchange/rest/AllParams;)Ljava/lang/String;
01-21 14:17:04.969: D/c_.x_.x_.r_.HttpTempla_(332): Executing GET request at https://btc-e.com/api/2/btc_usd/depth?raw
01-21 14:17:06.778: D/c_.x_.x_.r_.HttpTempla_(332): Response body: {"asks":[[16.15,154.326],[16.2,100],[16.249,2.60000073],[16.25,117.33275772],[16.3,100],[16.335,0.1],[16.336,0.1],[16.337,0.1],[16.338,0.1],[16.339,0.1],[16.34,0.1],[16.341,0.1],[16.342,0.1],[16.343,3],[16.345,3],[16.346,3],[16.347,3],[16.348,3],[16.35,116.69550068],[16.351,0.24691309],[16.389,3],[16.39,29.94],[16.398,2.988],[16.405,49.998],[16.406,0.16207],[16.488,2.4204403],[16.49,3.4447541],[16.499,2.5],[16.5,121.4793],[16.517,0.4549222],[16.52,1],[16.555,2.48811381],[16.567,3],[16.587,0.1],[16.588,5.04893817],[16.59,2.98799969],[16.597,3],[16.599,9.06817811],[16.6,9.12830864],[16.639,0.6],[16.65,0.33415909],[16.655,1.8],[16.68,2],[16.699,0.32727307],[16.7,23.29191645],[16.785,0.56],[16.795,66],[16.799,0.14967633],[16.8,20.9625645],[16.85,1.34998725],[16.89,1.1],[16.96,3.1],[16.973,1.99936668],[16.989,10],[16.99,24.42278885],[16.999,38.47172137],[17,21.38060851],[17.023,0.1],[17.119,0.1],[17.2,6.4255431],[17.4,0.1],[17.489,10],[17.499,2.0145],[17.5,4.35327999],[17.567,0.1],[17.6,0.1],[17.777,1],[17.888,1],[17.907,0.1],[17.974,99],[17.989,10],[17.999,1],[18,4.40819823],[18.132,0.1],[18.184,0.1],[18.489,10],[18.5,0.1],[18.555,0.1],[18.88,4.25998459],[18.884,0.1],[18.989,10],[19,13],[19.098,0.1],[19.159,0.1],[19.184,0.1],[19.265,25],[19.3,0.1],[19.465,25],[19.501,0.1],[19.555,0.1],[19.78,20],[19.951,0.1],[19.974,67.2],[19.99,2.27871],[19.997,14.999],[20,582],[20.1,3],[20.139,10],[20.14,43],[20.147,3.02344971],[20.164,7],[20.189,0.1],[20.53,7],[20.65,0.1],[20.701,0.1],[21,10.1],[21.001,45.233112],[21.2,0.1],[21.981,0.1],[22,60],[23,54.1788],[23.132,0.1],[23.979,0.1],[25,11.2],[25.598,0.1],[26.3,0.1],[29,0.1],[29.914,9.004],[30,2],[30.147,90],[33.132,0.1],[35,35],[45,11234.1],[50,5600],[55,0.1],[60,7854],[99.025,0.3],[99.99,0.1]],"bids":[[16.109,0.11554856],[16.105,0.11491068],[16.102,5],[16.051,0.11543391],[16.05,18.79],[16.042,0.11543391],[16.035,1],[16.03,5.79952598],[16.029,1.54176799],[16.028,0.166],[16,0.21543391],[15.986,3.9],[15.97,5.63426755],[15.925,0.3391],[15.909,0.4023],[15.894,0.16194],[15.873,0.6804],[15.799,10.99600534],[15.751,2.5],[15.75,9.6593],[15.7,0.1],[15.65,2.13661399],[15.641,8.23379844],[15.6,33.5259722],[15.56,12.49115873],[15.55,7.9956],[15.51,0.5],[15.501,3],[15.5,27.3],[15.464,0.10462807],[15.4,17.09],[15.36,0.1],[15.356,0.98756853],[15.35,26.7356],[15.34,0.1],[15.33,0.1],[15.32,0.1],[15.3,4.52470994],[15.253,7.71482297],[15.251,2.5],[15.25,0.48712898],[15.24,0.1316071],[15.233,1.39557576],[15.211,65.29571314],[15.21,8.06],[15.2,34.46153003],[15.15,5],[15.114,3.00514804],[15.11,8.5],[15.109,7.59583622],[15.101,151.86539879],[15.1,1.7994463],[15.08,10],[15.04,32],[15.024,0.157],[15.01,5.8],[15.001,13.39369557],[15,4.39419999],[14.991,0.18838],[14.986,0.107],[14.965,0.131],[14.95,4.26805762],[14.89,60.11142248],[14.84,1.63999],[14.79,12.1],[14.751,2.5],[14.71,5],[14.7,6.02384652],[14.599,1.67],[14.577,0.117],[14.519,0.1],[14.501,2.5],[14.5,228.83011357],[14.449,0.216],[14.426,1],[14.38,7.8156],[14.303,5],[14.251,2.5],[14.2,5.09542253],[14.159,0.225],[14.151,3.77799],[14.116,20],[14.042,2],[14.03,10],[14.001,2.5],[14,1.12080774],[13.861,0.157],[13.751,2.5],[13.737,0.169],[13.682,1],[13.61,9.03263435],[13.607,9.88986624],[13.595,0.142],[13.582,0.164],[13.575,0.213],[13.55,0.1],[13.502,0.1],[13.501,2.5],[13.5,14.36028515],[13.491,0.58991304],[13.483,8.5112146],[13.453,8],[13.429,0.93079911],[13.42,25],[13.4,8.3203651],[13.38,25.28980798],[13.319,0.217],[13.3,12],[13.251,2.5],[13.25,10],[13.223,8.3768],[13.22,28.08],[13.208,0.212],[13.203,10.28008784],[13.201,4.06],[13.2,8.02282265],[13.15,91.57334258],[13.137,2.50036002],[13.125,1],[13.1,10],[13.083,0.1],[13.067,0.1],[13.051,0.227],[13.05,0.9],[13.04,0.135],[13.026,0.1],[13.025,2.75455662],[13.019,0.181],[13.002,0.2],[13.001,2.5],[13,27.8199818],[12.99,20],[12.926,1.01900077],[12.913,0.289],[12.905,0.129],[12.9,1.442],[12.851,0.1],[12.845,0.1],[12.844,0.102],[12.822,25],[12.807,0.1],[12.804,0.1],[12.801,76.95415585],[12.8,2.35],[12.777,0.141],[12.755,0.189],[12.751,2.5],[12.72,10],[12.7,1
01-21 14:17:14.468: D/c_.x_.x_.ExchangeFacto_(332): Creating default exchange from class name
01-21 14:17:14.578: D/c_.x_.x_.r_.HttpTempla_(332): Executing GET request at https://btc-e.com/api/2/btc_usd/depth?raw
01-21 14:17:14.648: D/c_.x_.x_.r_.HttpTempla_(332): Response body:
01-21 14:17:14.648: W/System.err(332): java.lang.NullPointerException
01-21 14:17:14.658: W/System.err(332): at com.xeiam.xchange.btce.service.marketdata.polling.BTCEPollingMarketDataService.getFullOrderBook(BTCEPollingMarketDataService.java:88)
01-21 14:17:14.658: W/System.err(332): at com.veken0m.bitcoinium.OrderbookActivity.getOrderBook(OrderbookActivity.java:141)
01-21 14:17:14.658: W/System.err(332): at com.veken0m.bitcoinium.OrderbookActivity$OrderbookThread.run(OrderbookActivity.java:290)
01-21 14:17:15.579: I/ActivityManager(65): Displayed activity com.veken0m.bitcoinium/.OrderbookActivity: 1550 ms (total 1550 ms)

[timmolter]

Glad you got the logging working. That's great news. It looks like you made two calls for the full order book 10 seconds apart, and the second time, the response was empty. I'll have to think about that...

[veken0m]

Hmm, I hadn't noticed this before but if I make I make two calls for the BTC-E orderbook in a row the first call is always successful and the second call always fails. The pattern then repeats for any subsequent calls. The other exchanges don't have this problem. Bitstamp and VirtEx seem to fail at random intervals.

[timmolter]

They might have some throttling logic at BTC-E that causes this. You really shouldn't have to be calling it so often, right?

On another note, Matija, does your REST proxy provide a means of checking header response codes?

[veken0m]

Correct, I was just testing to see when it fails. Oddly enough, two BTC-E Orderbook calls in a row work fine on my phone.

[mmazi]

The REST proxy uses practically unchanged (just a bit simplified) HttpTemplate, which uses URLConnection that is really a HttpURLConnection, which has a getResponseCode() method. But HttpTemplate doesn't currently check the response code or provide means to access it through the rest framework.

[timmolter]

As a start, I added response code log at debug level to the REST proxy. Go ahead and use the latest snapshot jars, and you may get some more insight. We probably will want to use the return codes intelligently in some way and not allow null Strings to be passed out...

[veken0m]

First thing that stands out:

Request http status = -1 -> Failure
Response body: {Empty} -> Failure (returned even though empty...)

01-23 09:37:55.722: D/c_.x_.x_.ExchangeFacto_(276): Creating default exchange from class name
01-23 09:37:55.802: D/c_.x_.x_.r_.HttpTempla_(276): Executing GET request at https://cavirtex.com/api/CAD/orderbook.json
01-23 09:37:55.822: V/c_.x_.x_.r_.HttpTempla_(276): Request body =
01-23 09:37:55.822: V/c_.x_.x_.r_.HttpTempla_(276): Request headers = {Accept=application/json, Content-Type=application/x-www-form-urlencoded}
01-23 09:37:55.842: V/c_.x_.x_.r_.HttpTempla_(276): Header request property: key='User-Agent', value='XChange/1.4.0-SNAPSHOT JDK/6 AppleWebKit/535.7 Chrome/16.0.912.36 Safari/535.7'
01-23 09:37:55.852: V/c_.x_.x_.r_.HttpTempla_(276): Header request property: key='Content-Type', value='application/x-www-form-urlencoded'
01-23 09:37:55.862: V/c_.x_.x_.r_.HttpTempla_(276): Header request property: key='Accept', value='application/json'
01-23 09:37:55.862: V/c_.x_.x_.r_.HttpTempla_(276): Header request property: key='Accept-Charset', value='UTF-8'
01-23 09:37:55.991: D/c_.x_.x_.r_.HttpTempla_(276): Request http status = 200
01-23 09:37:56.001: I/global(276): Default buffer size used in BufferedReader constructor. It would be better to be explicit if an 8k-char buffer is required.
01-23 09:37:56.041: D/c_.x_.x_.r_.HttpTempla_(276): Response body: {"bids": [[0.010010000, 150.000000000], [0.121000000, 82.000000000], [0.010000000, 0.060000000], [0.000010000, 1000.000000000], [1.000000000, 100.000000000], [0.070000000, 1.000000000], [0.013100000, 700.000000000], [0.800000000, 100.000000000], [0.001000000, 3500.000000000], [0.050000000, 100.000000000], [0.160000000, 100.000000000], [0.010000000, 10.000000000], [0.610000000, 100.000000000], [0.250000000, 250.000000000], [0.010000000, 400.000000000], [2.750000000, 11.950000000], [2.800000000, 7.260000000], [1.000000000, 0.050000000], [4.000000000, 3.000000000], [3.000000000, 2.000000000], [4.500000000, 0.230000000], [4.250000000, 25.740000000], [5.440000000, 8.130000000], [4.000000000, 31.330000000], [3.750000000, 29.170000000], [4.500000000, 24.340000000], [3.500000000, 31.290000000], [7.200000000, 1.000000000], [8.600000000, 10.100000000], [4.000000000, 100.000000000], [5.000000000, 100.000000000], [3.000000000, 100.000000000], [2.000000000, 100.000000000], [1.173800000, 500.000000000], [1.000000000, 600.000000000], [0.000010000, 20000.000000000], [0.000500000, 1.000000000], [9.700000000, 6.510000000], [9.405000000, 10.000000000], [9.710000000, 4.190000000], [9.876540000, 1.000000000], [9.567890000, 1.000000000], [9.012340000, 1.000000000], [8.888880000, 1.000000000], [8.456780000, 1.000000000], [8.080800000, 1.000000000], [7.777770000, 1.000000000], [7.500000000, 1.000000000], [7.280000000, 1.000000000], [7.080000000, 1.000000000], [6.789000000, 1.000000000], [0.100000000, 2.000000000], [10.250000000, 5.840000000], [10.349980000, 13.720000000], [10.669970000, 8.720000000], [10.550000000, 15.000000000], [11.150000000, 20.000000000], [11.200000000, 1.000000000], [10.500000000, 2.000000000], [10.270000000, 6.000000000], [10.950000000, 4.000000000], [11.150000000, 4.200000000], [11.800000000, 50.000000000], [12.300000000, 100.000000000], [12.350000000, 20.000000000], [11.020000000, 6.500000000], [11.860000000, 7.000000000], [11.500000000, 2.000000000], [11.720000000, 22.000000000], [12.200000000, 19.660000000], [12.360000000, 10.000000000], [10.200000000, 5.000000000], [12.475000000, 10.000000000], [12.570000000, 3.000000000], [12.570000000, 0.650000000], [12.000000000, 0.600000000], [12.700000000, 2.830000000], [12.650000000, 2.000000000], [12.530000000, 10.000000000], [12.800000000, 4.900000000], [13.200000000, 37.000000000], [7.500000000, 103.560000000], [12.860000000, 3.000000000], [13.130000000, 1.000000000], [13.840000000, 1.670000000], [14.000000000, 41.140000000], [13.600000000, 50.000000000], [13.050000000, 50.000000000], [13.200100000, 10.000000000], [14.106510000, 6.370000000], [10.700000000, 2.000000000], [15.371230000, 10.230000000], [15.401000000, 0.800000000], [15.849900000, 2.210000000], [15.850000000, 10.300000000], [15.890000000, 2.060000000], [16.091000000, 37.830000000], [13.500000000, 18.340000000], [15.310000000, 10.300000000], [13.000900000, 10.000000000], [15.500000000, 10.000000000], [0.100000000, 0.010000000], [10.000000000, 26.550000000], [12.000000000, 85.000000000], [15.800000000, 514.540000000], [16.150000000, 2.000000000], [15.000000000, 48.000000000], [12.890900000, 6.000000000], [12.510900000, 1.500000000], [8.250000000, 0.040000000], [14.000000000, 1.480000000], [14.000000000, 29.000000000], [15.000000000, 12.000000000], [14.500000000, 13.000000000], [16.300000000, 1.000000000], [15.200000000, 947.000000000], [15.760000000, 21.600000000], [16.100000000, 50.000000000], [16.650000000, 513.400000000], [16.700000000, 3.000000000], [16.091010000, 0.490000000]], "asks": [[20.000000000, 19.880000000], [20.000000000, 4.500000000], [25.000000000, 1.490000000], [17.500000000, 30.000000000], [20.000000000, 20.000000000], [22.000000000, 15.940000000], [19.940000000, 5.000000000], [6100.000000000, 25.000000000], [20.000000000, 45.000000000], [20.000000000, 1.500000000], [19.000000000, 28.000000000], [20.000000000, 5.000000000], [20.000000000, 194.000000000], [18.000000000, 3.000000000], [18.240000000, 3.980000000], [18.479880000, 130.000000
01-23 09:39:02.941: D/c_.x_.x_.ExchangeFacto_(276): Creating default exchange from class name
01-23 09:39:03.011: D/c_.x_.x_.r_.HttpTempla_(276): Executing GET request at https://cavirtex.com/api/CAD/orderbook.json
01-23 09:39:03.011: V/c_.x_.x_.r_.HttpTempla_(276): Request body =
01-23 09:39:03.011: V/c_.x_.x_.r_.HttpTempla_(276): Request headers = {Accept=application/json, Content-Type=application/x-www-form-urlencoded}
01-23 09:39:03.132: D/dalvikvm(276): GC_FOR_MALLOC freed 11504 objects / 669536 bytes in 110ms
01-23 09:39:03.142: V/c_.x_.x_.r_.HttpTempla_(276): Header request property: key='User-Agent', value='XChange/1.4.0-SNAPSHOT JDK/6 AppleWebKit/535.7 Chrome/16.0.912.36 Safari/535.7'
01-23 09:39:03.142: V/c_.x_.x_.r_.HttpTempla_(276): Header request property: key='Content-Type', value='application/x-www-form-urlencoded'
01-23 09:39:03.142: V/c_.x_.x_.r_.HttpTempla_(276): Header request property: key='Accept', value='application/json'
01-23 09:39:03.152: V/c_.x_.x_.r_.HttpTempla_(276): Header request property: key='Accept-Charset', value='UTF-8'
01-23 09:39:03.172: D/c_.x_.x_.r_.HttpTempla_(276): Request http status = -1
01-23 09:39:03.212: I/global(276): Default buffer size used in BufferedInputStream constructor. It would be better to be explicit if an 8k buffer is required.
01-23 09:39:03.212: D/c_.x_.x_.r_.HttpTempla_(276): Response body:
01-23 09:39:03.212: W/System.err(276): java.lang.NullPointerException
01-23 09:39:03.222: W/System.err(276): at com.xeiam.xchange.virtex.service.marketdata.polling.VirtExPollingMarketDataService.getFullOrderBook(VirtExPollingMarketDataService.java:94)

[gary-rowe]

Just chiming in, you get a -1 from HttpURLConnection when the response is not valid HTTP (see http://javasourcecode.org/html/open-source/jdk/jdk-6u23/java/net/HttpURLConnection.html#getResponseCode%28%29). It may be that virtex are using their own RMI approach that mimics HTTP.

[timmolter]

I looked into this last night a bit more. I think our Http proxy class (HttpTemplate.java) needs to redesigned. While looking into things, I also found that the underlying httpURLConnections should be handled differently for Android. I'm thinking of using this ultra-lightweight lib inside of XChange-core, at least pieces we need of it: http://code.google.com/p/basic-http-client/ That, along with a way to tell XChange that it's running on Android or not.

Michael, for now, I'd say catch the null pointers and handle them appropriately. I think in version 1.5.0, we'll have a more robust http interface with better error handling, but for the short term, it's not possible.

[veken0m]

Tim, if it was a simple fix I would say go for it but if it's too much of a hassle it's not really worth
fixing right now considering it might only affect a small portion of Bitcoinium users (~3%) and that number will drop as people upgrade their devices. By handling the exception, the app is still perfectly functional for the users affected. I'd rather not waste any more time with this issue.

[timmolter]

Sounds good. I'll close this issue now. We'll clean it up in a later version.