Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create parser for list of all existing wikis #1

Closed
8 tasks done
Retailed opened this issue Dec 14, 2020 · 0 comments · Fixed by #10
Closed
8 tasks done

Create parser for list of all existing wikis #1

Retailed opened this issue Dec 14, 2020 · 0 comments · Fixed by #10
Assignees

Comments

@Retailed
Copy link
Collaborator

Retailed commented Dec 14, 2020

Description

To fetch all the Scribunto modules in all Wikimedia pages, first thing to do is to get list of them. That can be done by parsing an existing page on Meta-wiki. For further usage, parsed pages should be saved in text file.

The update (15.12.2020)

For fetching additional information we need to know names of different wikis in database too. So it is reasonable to switch to fetching this info from 'meta' wiki database copy, as written in here

But these tables don't have update time property, as they are just copies, which are not updated per se, the new copies are just loaded instead of old ones. So it makes sense to look at time of creation to check for updates.

Tasks

  • Parse existing wiki links from the page
  • Save them in text file, one line - one link
  • Add checks for page parsing (page unavailable, page changed...)
  • Add check if page have been updated recently (api request?)
  • Move to fetching info from database copies
  • Save request results as csv
  • Make "last update" checker
  • Put updater to cron
@Retailed Retailed self-assigned this Dec 14, 2020
@Retailed Retailed added the in progress Assigned tasks in progress label Dec 14, 2020
@Retailed Retailed linked a pull request Dec 17, 2020 that will close this issue
@Retailed Retailed removed the in progress Assigned tasks in progress label Dec 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant