Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Missing tasks with low semaphore value #2

Closed
mccarthyryanc opened this issue Sep 22, 2021 · 2 comments · Fixed by #3
Closed

Missing tasks with low semaphore value #2

mccarthyryanc opened this issue Sep 22, 2021 · 2 comments · Fixed by #3

Comments

@mccarthyryanc
Copy link
Collaborator

I have a local server for ept data, running a query like:

query = ept.EPT(
    'http://localhost/data/ept/',
    bounds=ept.Bounds(774163, 3482814, 0, 774263, 3482914, 1000)
)
data = query.data()

The server shows the correct GET requests in the logs:

192.168.0.1 - - [22/Sep/2021:19:16:06 +0000] "GET /data/ept/ept.json HTTP/1.1" 200 2959 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:10 +0000] "GET /data/ept/ept-hierarchy/0-0-0-0.json HTTP/1.1" 200 529057 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:10 +0000] "GET /data/ept/ept-hierarchy/8-107-146-127.json HTTP/1.1" 200 8577 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:11 +0000] "GET /data/ept/ept-data/1-0-1-1.laz HTTP/1.1" 200 6963 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:11 +0000] "GET /data/ept/ept-data/2-1-2-2.laz HTTP/1.1" 200 5246 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:11 +0000] "GET /data/ept/ept-data/3-3-4-4.laz HTTP/1.1" 200 2171 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:11 +0000] "GET /data/ept/ept-data/4-6-9-8.laz HTTP/1.1" 200 1681 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:11 +0000] "GET /data/ept/ept-data/1-0-1-0.laz HTTP/1.1" 200 100090 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:11 +0000] "GET /data/ept/ept-data/0-0-0-0.laz HTTP/1.1" 200 106405 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:11 +0000] "GET /data/ept/ept-data/3-3-4-3.laz HTTP/1.1" 200 199839 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:12 +0000] "GET /data/ept/ept-data/4-6-9-7.laz HTTP/1.1" 200 222579 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:12 +0000] "GET /data/ept/ept-data/2-1-2-1.laz HTTP/1.1" 200 208395 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:12 +0000] "GET /data/ept/ept-data/12-1713-2349-2043.laz HTTP/1.1" 200 349128 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:12 +0000] "GET /data/ept/ept-data/12-1715-2349-2043.laz HTTP/1.1" 200 307874 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:12 +0000] "GET /data/ept/ept-data/12-1715-2348-2043.laz HTTP/1.1" 200 315234 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:12 +0000] "GET /data/ept/ept-data/12-1714-2348-2043.laz HTTP/1.1" 200 360543 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:12 +0000] "GET /data/ept/ept-data/12-1714-2349-2043.laz HTTP/1.1" 200 356531 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:12 +0000] "GET /data/ept/ept-data/5-13-18-15.laz HTTP/1.1" 200 340836 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:12 +0000] "GET /data/ept/ept-data/12-1713-2348-2043.laz HTTP/1.1" 200 458484 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:12 +0000] "GET /data/ept/ept-data/6-26-36-31.laz HTTP/1.1" 200 454641 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:12 +0000] "GET /data/ept/ept-data/7-53-73-63.laz HTTP/1.1" 200 493448 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:12 +0000] "GET /data/ept/ept-data/11-857-1174-1021.laz HTTP/1.1" 200 546097 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:13 +0000] "GET /data/ept/ept-data/10-428-587-510.laz HTTP/1.1" 200 605378 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:13 +0000] "GET /data/ept/ept-data/8-107-146-127.laz HTTP/1.1" 200 674198 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:13 +0000] "GET /data/ept/ept-data/11-856-1174-1021.laz HTTP/1.1" 200 629387 "-" "Python/3.8 aiohttp/3.7.4.post0"
192.168.0.1 - - [22/Sep/2021:19:16:13 +0000] "GET /data/ept/ept-data/9-214-293-255.laz HTTP/1.1" 200 652445 "-" "Python/3.8 aiohttp/3.7.4.post0"

However, the length on data changes with the value that is set for the task pool semaphore. If the value is 10 I get something around 12-14 task results, but if I set the value to 100 I get all 23 task results.
@hobu
Copy link
Owner

hobu commented Sep 22, 2021

Some kind of bug in the semaphore stuff. This code was not ever put into production. Congrats on being the first 😄

@mccarthyryanc
Copy link
Collaborator Author

I am happy to take look and submit a PR if I can figure it out.

@mccarthyryanc mccarthyryanc mentioned this issue Sep 22, 2021
Merged
@hobu hobu closed this as completed in #3 Sep 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Adding fix for non-unique ids in task cache dict mccarthyryanc/ept-python
2 participants
@hobu @mccarthyryanc