Inefficient caching of FdsnNetExtender.extend()

[andres-h]

FdsnNetExtender.extend(self, net, date_string) has lru_cache(maxsize=1000), but since date_string is different most of the time, caching seems to be inefficient. In any case, I can observe urls like http://www.fdsn.org/ws/networks/1/query?fdsn_code=3E being downloaded hundreds of times. Sometimes this causes an exception, which seems to be the reason of incomplete statistics at GFZ.

Maybe date_string should be reduced to year (two different temporary networks with the same code never exist in same year?). Alternatively I would suggest caching the result of urlopen(request).

[jschaeff]

Nice spotting. I'll fix fdsnnetextender.

[jschaeff]

fdsnnetextender 3.3.0 implements caching of the FDSN request only.

Tests are OK.

Could you upgrade only this package ? If it's working alright, then we advertise the new version to everyone, it's an important fix.

[andres-h]

Seems to work OK now.

[jschaeff]

Let it run during the week end. On monday, if everything is fine, I'll make an announce for everyone to update.

Thank you for your help !

[jschaeff]

@andres-h can you confirm that this fix is improving the situation ?

[andres-h]

@andres-h can you confirm that this fix is improving the situation ?

Absolutely. Now after 5 days it is:

$ grep ^REQUESTING send_stats.log | sort | uniq -c | sort -n -r | head
5 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=ZW
5 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=ZS
5 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=ZE
5 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=Z3
5 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=YZ
5 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=XN
5 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=9J
5 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=9H
5 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=9C
5 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=8H

while previously:

$ grep ^REQUESTING send_stats.log.old | sort | uniq -c | sort -n -r | head
931 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=Z3
687 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=3E
644 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=5C
613 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=YU
397 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=YZ
264 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=XO
258 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=Y4
229 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=ZA
229 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=XE
151 REQUESTING http://www.fdsn.org/ws/networks/1/query?fdsn_code=Y9

[jschaeff]

Thank you for your help !