Can't uncompress snapshot with DonorsChoose.org data

[robertlyson]

Hi,

Thank you very much for the example with DonorsChoose data, it's awesome!

I wanted to run the example on my local pc, unfortunately I have problem with downloaded snapshot - I can't unpack it.
I tried with winrar and 7-zip, but both failed. Error is always the same:

What is quite interesting to me, is fact that when I'm downloading file, the size is 7.5GB:

File downloads fine, but its size is different than expected:

Only 4.95GB.

OS: windows 10
File System: NTFS
Free space before unpacking: 10 GB

Any help much appreciated.
Thank you.

[markwalkom]

I'm just waiting to download this, then I'll try it out.

[asawariS]

@robertlyson - @markwalkom (thanks Mark!) and I are looking into this. Will keep you posted.

[asawariS]

@robertlyson I am not able to reproduce this - still looking into it. Sorry about that the hiccups here

In the mean time, do you mind trying the Python script version of getting data indexed into Elasticsearch? At least that way you can try the example out and play with data. I will also find a way to perhaps share the uncompressed version of the index with you.

Finally, it may not matter. But, the data uncompresses to about 11 GB - so if you only have 10 GB free space pre-unpacking, that may be a problem.

Asawari

[robertlyson]

@asawariS, @markharwood first of all, thanks for paying attention to this issue.

I forgot that I can index data with python script, thanks for reminder :)
Unfortunately when I tried donorschoose_process_data.py I got following error:

D:\Temp\kibana sample>python donorschoose_process_data.py
Traceback (most recent call last):
  File "donorschoose_process_data.py", line 47, in <module>
    donations = pd.read_csv('./data/opendata_donations.csv', index_col=None)
  File "C:\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 474, in parse
r_f
    return _read(filepath_or_buffer, kwds)
  File "C:\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 260, in _read

    return parser.read()
  File "C:\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 721, in read
    ret = self._engine.read(nrows)
  File "C:\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 1170, in read

    data = self._reader.read(nrows)
  File "pandas\parser.pyx", line 769, in pandas.parser.TextReader.read (pandas\p
arser.c:7544)
  File "pandas\parser.pyx", line 791, in pandas.parser.TextReader._read_low_memo
ry (pandas\parser.c:7784)
  File "pandas\parser.pyx", line 844, in pandas.parser.TextReader._read_rows (pa
ndas\parser.c:8401)
  File "pandas\parser.pyx", line 831, in pandas.parser.TextReader._tokenize_rows
 (pandas\parser.c:8275)
  File "pandas\parser.pyx", line 1742, in pandas.parser.raise_parser_error (pand
as\parser.c:20691)
pandas.parser.CParserError: Error tokenizing data. C error: *Expected 45 fields i
n line 315850, saw 46*

I have python 3.4 and anaconda 2.3.0.
Do you believe it's related with python or anaconda version?

[asawariS]

@robertlyson

Thanks for bring it to my attention. Looks like the data might have been updated recently, and there's some format change in the header. After you unzip the 3 CSV files, can you make sure the header line and 1st row row of data have a line separator between them. Additionally make sure that there are no line separators between the column names in the header. I noticed that the the line separator was before "donation_message" column name instead of after it in the opendata_donations.csv file. Similar case with the opendata_resources.csv file - this caused the pandas read csv function to expect the incorrect number of columns.

I have also updated the Python code a little. So, make sure you grab the updated version of the scripts from the Scripts - Python folder.

Give it a go, and let me know if it works.

[robertlyson]

@asawariS updated version of the python script did the job 👍

Thank you guys for your support.

[changyh]

I am run it on mac with python 3.5 and got the following errors on
python3 donorschoose_process_data.py
.............
1290000
PUT http://localhost:9200/donorschoose/donation/4751214b9cc2f47bc1d26e98cead7946 [status:N/A request:10.214s]
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 376, in _make_request
httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 378, in _make_request
httplib_response = conn.getresponse()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1174, in getresponse
response.begin()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 282, in begin
version, status, reason = self._read_status()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 243, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socket.py", line 571, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/elasticsearch/connection/http_urllib3.py", line 78, in perform_request
response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 609, in urlopen
_stacktrace=sys.exc_info()[2])
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/util/retry.py", line 222, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/packages/six.py", line 310, in reraise
raise value
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 559, in urlopen
body=body, headers=headers)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 380, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 308, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=9200): Read timed out. (read timeout=10)
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 376, in _make_request
httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 378, in _make_request
httplib_response = conn.getresponse()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1174, in getresponse
response.begin()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 282, in begin
version, status, reason = self._read_status()
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 243, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/socket.py", line 571, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/elasticsearch/connection/http_urllib3.py", line 78, in perform_request
response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 609, in urlopen
_stacktrace=sys.exc_info()[2])
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/util/retry.py", line 222, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/packages/six.py", line 310, in reraise
raise value
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 559, in urlopen
body=body, headers=headers)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 380, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 308, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=9200): Read timed out. (read timeout=10)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "donorschoose_process_data.py", line 143, in
es.index(index=index_name, doc_type=doc_name, id=thisDonation['donationid'], body=thisDoc)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
return func(_args, params=params, *_kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/elasticsearch/client/init.py", line 261, in index
_make_path(index, doc_type, id), params=params, body=body)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/elasticsearch/transport.py", line 307, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/elasticsearch/connection/http_urllib3.py", line 86, in perform_request
raise ConnectionTimeout('TIMEOUT', str(e), e)
elasticsearch.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(host='localhost', port=9200): Read timed out. (read timeout=10))

Then:
yhs-MacBook-Pro:elk_donorschoose yh$ curl -XGET localhost:9200/donorschoose/_count -d '{

"query": {
    "match_all": {}
}

}'
{"count":1293665,"_shards":{"total":5,"successful":5,"failed":0}}

On the import dashboard: donorschoose_dashboard.json
I got the following errors on kibana:

Error: Could not locate that index-pattern (id: usfec*)
at applyESResp (http://localhost:5601/index.js?_b=7562:89411:36)
at wrappedCallback (http://localhost:5601/index.js?_b=7562:20893:81)
at http://localhost:5601/index.js?_b=7562:20979:26
at Scope.$eval (http://localhost:5601/index.js?_b=7562:22022:28)
at Scope.$digest (http://localhost:5601/index.js?_b=7562:21834:31)
at Scope.$apply (http://localhost:5601/index.js?_b=7562:22126:24)
at done (http://localhost:5601/index.js?_b=7562:17661:45)
at completeRequest (http://localhost:5601/index.js?_b=7562:17875:7)
at XMLHttpRequest.xhr.onreadystatechange (http://localhost:5601/index.js?_b=7562:17814:11)

[asawariS]

@changyh

Issue # 1 - increase the timeout period in the index step on line 143 by adding the timeout parameter. You can start with 30 and see if that fixes request timed out error.
es.index(index=index_name, doc_type=doc_name, id=thisDonation['donationid'], body=thisDoc, timeout= 30)

Issue # 2- By mistake, I had included an additional visualization (that required the usfec* index) during the export step. I have update the dashboard JSON file to get rid of that viz. Try importing the updated dashboard JSON again - should work without errors this time.

Let me know how these 2 suggestions work out for you.

[changyh]

@asawariS

Thanks for the help,

I did delete the usfec portion of the dashboard json by myself, so I can saw the dashboard,...so we don't need this part,...

[asawariS]

Awesome. Glad it works!

[changyh]

@asawariS ,

Do you think the following python error is course by time out on write:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 376, in _make_request
httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:
.....

[asawariS]

@changyh The full error stack in your previous note seemed to indicate a timeout error on the data indexing step. What command in the script is the above error attributed to?

[changyh]

The command is:

python3 donorschoose_process_data.py

after long run, then I got the above errors,...

By the way in the example do we use Logstash, where do we use it?

Thanks,