genealogics.org partially scraped

[marfox]

Out of the claimed 668,261 items:

• the first run yielded 447,045 items, then timed out;
• the second run yielded 266,492, then timed out as well.

For the Development corpus milestone, let's move on and combine the outputs of the runs.

Then, we can leave this issue open, assign it to the Production Corpus milestone, and re-run the scraper until we have all the data.

[tfmorris]

None of the crowdsourced secondary data from genealogics should be used anyway. You would have to follow the citations ("sources" in genealogy terms) back to the primary sources to get anything usable.

[marfox]

Thanks @tfmorris for the comment.
That source was considered reliable for 2 reasons:

1. it is included in the mix'n'match catalog list
2. Identifiers have their own Wikidata property

Would you mind moving this conversation to a new section in https://www.wikidata.org/wiki/Wikidata:Requests_for_comment/Semi-automatic_Addition_of_References_to_Wikidata_Statements ?

UPDATE: just saw your edit on https://meta.wikimedia.org/wiki/Grants_talk:IEG/StrepHit:_Wikidata_Statements_Validation_via_References/Timeline , I'll link that into the RFC.

[tfmorris]

Perhaps this statement from the genealogics home page has something to do with your inability to mine the site:

This site employs an automatic process to detect data mining. While reasonable private use of this facility is encouraged, data mining will result in denial of access.