Error after downloading around 30 files.

[ghost]

After running I get this error. This has only started happening in the last few days. My guess is it might have to do with CloudFlare blocking the multiple requests in such a short time. A work around might be adding a counter that pauses the download for a certain amount of time after so many. I would try myself if I knew python.

Traceback (most recent call last):
File "C:\Python27\Lib\multiprocessing\process.py", line 258, in _bootstrap
self.run()
File "C:\Python27\Lib\multiprocessing\process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "E:\New folder\e621dl-master alt\e621dl-master\lib\downloader.py", line 41, in download_monitor
update_progress(len(managed_list), total_items)
File "", line 2, in len
File "C:\Python27\Lib\multiprocessing\managers.py", line 758, in _callmethod
conn.send((self._id, methodname, args, kwds))
IOError: [Errno 232] The pipe is being closed
Traceback (most recent call last):
File "e621dl.py", line 125, in
downloader.multi_download(download_list, cpu_count())
File "E:\New folder\e621dl-master alt\e621dl-master\lib\downloader.py", line 78, in multi_download
work.get(0xFFFF)
File "C:\Python27\Lib\multiprocessing\pool.py", line 558, in get
raise self._value
IOError: [Errno socket error] [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

I can supply Debug if you want it.

[Wulfre]

It may have been a temporary error with e621, or your own connection may just be able to download the images faster than mine. I just downloaded 500 images with no issues.

It also looks like you are using an older version of the script, because you are using Python 2 still. Could you use the latest version and tell me if the problem still occurs? You are going to need to download Python 3.

[ghost]

I've updated my python and used the latest e621dl, still getting the error (though it did download around 300 this time), but it probable is my end as when I use cmd to ping static1.e621.net quite a few "request timed out"s appear. Will look more into it (its probable either shit wifi or my ISP being a dick.).

[ghost]

output.txt

This is the debug if it is any help. It seems that the downloads freeze up because the images are created but are 0 Bytes in size.

[Wulfre]

I changed the module that I am using for downloading images from spoofing urllib to requests, which is more simple. Grab the master files and see if that helps you at all.

[ghost]

Seems to work better, still some half download files and an error did come up.
Though these problems might have been caused by my internet connection.
Here is the Traceback anyways.

Traceback (most recent call last):
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\process.py", line 249, in _bootstrap
self.run()
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "E:\New folder\e621dl-test\e621dl-master\lib\downloader.py", line 38, in download_monitor
update_progress(len(managed_list), total_items)
File "", line 2, in len
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\managers.py", line 756, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\connection.py", line 280, in _send_bytes
ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)
BrokenPipeError: [WinError 232] The pipe is being closed
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests\packages\urllib3\response.py", line 298, in _error_catcher
yield
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests\packages\urllib3\response.py", line 380, in read
data = self._fp.read(amt)
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\http\client.py", line 449, in read
n = self.readinto(b)
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\http\client.py", line 493, in readinto
n = self.fp.readinto(b)
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\ssl.py", line 1002, in recv_into
return self.read(nbytes, buffer)
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\ssl.py", line 865, in read
return self._sslobj.read(len, buffer)
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\ssl.py", line 625, in read
v = self._sslobj.read(len, buffer)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests\models.py", line 719, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests\packages\urllib3\response.py", line 432, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests\packages\urllib3\response.py", line 397, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\contextlib.py", line 100, in exit
self.gen.throw(type, value, traceback)
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests\packages\urllib3\response.py", line 316, in _error_catcher
raise ProtocolError('Connection broken: %r' % e, e)
requests.packages.urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)", ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\pool.py", line 44, in mapstar
return list(map(*args))
File "E:\New folder\e621dl-test\e621dl-master\lib\downloader.py", line 50, in single_download
for chunk in data:
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests\models.py", line 722, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)", ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "e621dl.py", line 130, in
downloader.multi_download(download_list, cpu_count())
File "E:\New folder\e621dl-test\e621dl-master\lib\downloader.py", line 69, in multi_download
work.get(65535)
File "C:\Users\ \AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\pool.py", line 608, in get
raise self._value
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)", ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

[Wulfre]

I put out another update that cleaned up a lot of inefficient multiprocessing stuff that was left over from when I first forked. If that doesn't fix the problem, then it is either your internet, or another error that I am missing. My internet isn't the fastest, so there are some cases which I just can't test. Maybe your case is what happens when you download images too fast.

[ghost]

That seems to have fixed it. I had no problems downloading over 4000 files.

On a side note while you have been doing that I added some options to the config for filtering posts.
I dont know how well I coded it as I dont know a massive amount, but I have made it so it can filter posts by file type, rating and score as well as tags (more info in const.py).

More filters.zip

[Wulfre]

Glad to know that its working for you now! If you find anything else wrong, let me know, I love making improvements on this.

To avoid confusion between this issue and your side note, I am going to create a new issue (#4).