Query timeout limit reached while updating German Nouns

[shashank-iitbhu]

Terms

• I have searched all open bug reports
• I agree to follow Scribe-Data's Code of Conduct

Behavior

Description

(scribedev) shashankmittal@ShashanksLaptop Scribe-Data % python3 src/scribe_data/extract_transform/wikidata/update_data.py '["German"]' '["nouns", "verbs"]' 
Data updated:   0%|                                                                                                                   | 0/2 [00:00<?, ?dirs/s]Querying and formatting German nouns
Data updated:   0%|                                                                                                                   | 0/2 [01:00<?, ?dirs/s]
Traceback (most recent call last):
  File "/Users/shashankmittal/Documents/Developer/scribe/Scribe-Data/src/scribe_data/extract_transform/wikidata/update_data.py", line 141, in <module>
    results = sparql.query().convert()
  File "/opt/anaconda3/envs/scribedev/lib/python3.10/site-packages/SPARQLWrapper/Wrapper.py", line 1196, in convert
    return self._convertJSON()
  File "/opt/anaconda3/envs/scribedev/lib/python3.10/site-packages/SPARQLWrapper/Wrapper.py", line 1059, in _convertJSON
    json_str = json.loads(self.response.read().decode("utf-8"))
  File "/opt/anaconda3/envs/scribedev/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/opt/anaconda3/envs/scribedev/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/anaconda3/envs/scribedev/lib/python3.10/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Invalid control character at: line 320797 column 115 (char 6713171)

Query builder Link

The query time limit is reached that's why results = sparql.query().convert() in update_data.py is throwing json.decoder.JSONDecodeError due to Invalid control character at: line 320797 column 115 (char 6713171) in sparql.query().response as it contains the timeout error logs.

Suggested Changes

• Considered splitting SPARQL query into smaller queries, such as one query for nouns and another for pronouns, or querying for singular and plural forms separately.
• Still got Query timeout limit reached error as total number of nouns and pronouns for German are 165869. Verified here.
• Use LIMIT and OFFSET to split into multiple queries.

[shashank-iitbhu]

This can be reproduced by running:
python3 src/scribe_data/extract_transform/wikidata/update_data.py '["German"]' '["nouns", "verbs"]'
@andrewtavis Are you able to reproduce this issue?
If so, I can open a PR with the proposed changes.

[andrewtavis]

I can confirm on my end, @shashank-iitbhu:

json.decoder.JSONDecodeError: Invalid control character at: line 320797 column 115 (char 6713171)

Two questions to decide on this:

• Could you do a query of just nouns and just pronouns and let us know what the percentage of the total each is?
• For LIMIT and OFFSET I'm a bit worried that it might not work as we'd need to compute the full result in order to then offset for the ones we want?
  • This would maybe be a solution if the result JSON was too large, but then this issue is query time
  • I can check with folks at work on this 😇

All in all great that you figured this out and suggested solutions! As you can see by the verbs queries, this is not the first time that this has happened 😅

[andrewtavis]

Confirmed from the Wikidata team that splitting based on nouns and proper nouns would be the initial path forward, but offset could work if it continues to be problematic :)

[andrewtavis]

Note that I just tried to query only all the singulars of only the nouns, not the proper nouns, and it's still failing. At this point it might make sense to use LIMIT and OFFSET.

[andrewtavis]

CC @daveads, do you want to write in here so I can assign this issue to you?

[daveads]

yup @andrewtavis