"Accept-Encoding: gzip, deflate" header is ignored?

[giuseppec]

I am using httr for an api call in my package and I want to do the following call in R using httr:

curl "http://www.openml.org/api/v1/task/list/limit/3000?api_key=c1994bdb7ecb3c6f3c8f3b35f4b47f1f" -H "Accept-Encoding: gzip, deflate"

The line above returns a gzip compressed string. But when I use the httr R-package it seems that "Accept-Encoding: gzip, deflate" is ignored because the returned body is not compressed :

library(httr)
content = GET(url = "http://www.openml.org/api/v1/task/list/limit/3000?api_key=c1994bdb7ecb3c6f3c8f3b35f4b47f1f", add_headers(`Accept-Encoding` = "gzip, deflate"))

[hadley]

How can you tell that the returned body is not compressed?

[giuseppec]

Short explanation:

Without the Accept-Encoding: "gzip, deflate" header, the server returns an uncompressed xml by default. However, I always receive the uncompressed content, no matter if I use add_headers(Accept-Encoding = "gzip, deflate") or not in the GET request, e.g.

res1 = GET(url = "http://www.openml.org/api/v1/task/list/limit/1?api_key=c1994bdb7ecb3c6f3c8f3b35f4b47f1f", 
  add_headers(`Accept-Encoding` = "gzip, deflate"))
res2 = GET(url = "http://www.openml.org/api/v1/task/list/limit/1?api_key=c1994bdb7ecb3c6f3c8f3b35f4b47f1f")

# I would expect that res1$content contains the compressed xml and res2$content the uncompressed xml, but:
identical(res1$content, res2$content)
# TRUE

rawToChar(res2$content)

[1] "<oml:tasks xmlns:oml=\"http://openml.org/openml\">\n    <oml:task>\n    <oml:task_id>1</oml:task_id>\n    <oml:task_type>Supervised Classification</oml:task_type>\n    <oml:did>1</oml:did>\n    <oml:name>anneal</oml:name>\n    <oml:status>active</oml:status>\n    <oml:format>ARFF</oml:format>\n          <oml:input name=\"estimation_procedure\">1</oml:input>\n          <oml:input name=\"evaluation_measures\">predictive_accuracy</oml:input>\n          <oml:input name=\"source_data\">1</oml:input>\n          <oml:input name=\"target_feature\">class</oml:input>\n              <oml:quality name=\"MajorityClassSize\">684.0</oml:quality>\n          <oml:quality name=\"MaxNominalAttDistinctValues\">10.0</oml:quality>\n          <oml:quality name=\"MinorityClassSize\">0.0</oml:quality>\n          <oml:quality name=\"NumberOfClasses\">6.0</oml:quality>\n          <oml:quality name=\"NumberOfFeatures\">39.0</oml:quality>\n          <oml:quality name=\"NumberOfInstances\">898.0</oml:quality>\n          <oml:quality name=\"NumberOfInstancesWithMissingValues\">0.0</oml:quality>\n          <oml:quality name=\"NumberOfMissingValues\">0.0</oml:quality>\n          <oml:quality name=\"NumberOfNumericFeatures\">6.0</oml:quality>\n          <oml:quality name=\"NumberOfSymbolicFeatures\">33.0</oml:quality>\n              <oml:tag>basic</oml:tag>\n          <oml:tag>study_1</oml:tag>\n          <oml:tag>study_7</oml:tag>\n          <oml:tag>under100k</oml:tag>\n          <oml:tag>under1m</oml:tag>\n      </oml:task>\n  </oml:tasks>\n"

Works with CURL:

1. Without the Accept-Encoding header, a string containing the XML is returned:

curl "http://www.openml.org/api/v1/task/list/limit/1?api_key=c1994bdb7ecb3c6f3c8f3b35f4b47f1f"

Result:

<oml:tasks xmlns:oml="http://openml.org/openml">
    <oml:task>
    <oml:task_id>1</oml:task_id>
    <oml:task_type>Supervised Classification</oml:task_type>
    <oml:did>1</oml:did>
    <oml:name>anneal</oml:name>
    <oml:status>active</oml:status>
    <oml:format>ARFF</oml:format>
          <oml:input name="estimation_procedure">1</oml:input>
          <oml:input name="evaluation_measures">predictive_accuracy</oml:input>
          <oml:input name="source_data">1</oml:input>
          <oml:input name="target_feature">class</oml:input>
              <oml:quality name="MajorityClassSize">684.0</oml:quality>
          <oml:quality name="MaxNominalAttDistinctValues">10.0</oml:quality>
          <oml:quality name="MinorityClassSize">0.0</oml:quality>
          <oml:quality name="NumberOfClasses">6.0</oml:quality>
          <oml:quality name="NumberOfFeatures">39.0</oml:quality>
          <oml:quality name="NumberOfInstances">898.0</oml:quality>
          <oml:quality name="NumberOfInstancesWithMissingValues">0.0</oml:quality>
          <oml:quality name="NumberOfMissingValues">0.0</oml:quality>
          <oml:quality name="NumberOfNumericFeatures">6.0</oml:quality>
          <oml:quality name="NumberOfSymbolicFeatures">33.0</oml:quality>
              <oml:tag>basic</oml:tag>
          <oml:tag>study_1</oml:tag>
          <oml:tag>study_7</oml:tag>
          <oml:tag>under100k</oml:tag>
          <oml:tag>under1m</oml:tag>
      </oml:task>
  </oml:tasks>

(this is basically the same content as from rawToChar(res2$content))

1. With the Accept-Encoding, a gzip compressed string is returned:

curl "http://www.openml.org/api/v1/task/list/limit/1?api_key=c1994bdb7ecb3c6f3c8f3b35f4b47f1f" -H "Accept-Encoding: gzip, deflate"

Result (gzip compressed):

▼      ♥¡ö¦n┌@►à´¹¶╚☼►@T)A─RÈ
h╠┴↑♦ı¥Nì■¢gÓÓs►L←l§]½Î¼¡Ë└¨¦Å┼óUtìñh»ñ#S♣▲4♂▄fÞÖt→║¿£§(â├l;b↕▲Ào@àÍ«◄|4¹,»∟JJâ▬
áêÙ♫± ┐¼ïe┌╚%²ÄöÙÚþ½QKÚ┤Cý3Ì▲¡&♥ÛĨ←┼dìÓº↑P‼╩xt&î╠■@g↕▲â^í¹¥NäfäÙ╦³ï6ÿ▄\F©7±[5óA
Loª ╚°I³³@±o3Õ6█♂â¨?öxGGóOÚ┬ÿùÁ^YÁ♂ÜLÄÉ>h♀e¥☻Ob{Ûö♂6=¤AÍ┼°¶ÐùÒó`$║±h¶rèL▼ê·├▒Ú¶U
<╔Ì☺▬ùaO┼♣

Is this enough as minimal example?

[hadley]

I'm pretty sure httr just uncompresses it for you, so you'll need to provide some evidence that the transport itself is not compressed.

[giuseppec]

You are absolutely right. I tested it with a network sniffer; the received file is compressed. Thx. I'll close.