Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Godot docs to include v4.2 and fix older version scraping #2201

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

jessevanherk
Copy link

@jessevanherk jessevanherk commented Apr 22, 2024

This adds docs for Godot v4.2, and does some fixups for scraping older godot releases from the upstream website.

Godot 4.2 was released on Nov 2023: https://godotengine.org/article/godot-4-2-arrives-in-style/

Since v3.5, godot upstream docs have used some new HTML layouts and file structures. Since older versions still need to be scrapable, this introduces another set of versioned filters for godot.

This change also makes small changes to the filters to handle the current website markup for the previous godot doc versions.

The @GDscript and @globalscope entries can't currently be browsed, because of an encoding mismatch between the frontend and backend; I've identified a possible fix in #2200 .

checklist

If you're updating existing documentation to its latest version, please ensure that you have:

  • Updated the versions and releases in the scraper file
  • Ensured the license is up-to-date
  • Ensured the icons and the SOURCE file in public/icons/your_scraper_name/ are up-to-date if the documentation has a custom icon
  • Ensured self.links contains up-to-date urls if self.links is defined
  • Tested the changes locally to ensure:
    • The scraper still works without errors
    • The scraped documentation still looks consistent with the rest of DevDocs
    • The categorization of entries is still good

QA notes

I generated all of the godot doc versions that the scraper supports, and diffed them against the existing packaged versions to ensure as close of a match as possible.

Command to generate a local copy of the godot 4.2 docs:

bundle exec thor docs:generate godot@4.2

Spot checks show that the main differences are:

  • the attribution statement, which I updated based on the attribution listed on all of the godot public docs going back to 2.1.
  • class_@gdscript and class_@globalscope filenames are escaped with %40 instead of @. I have no idea how the previous packager got those files to be @ symbols, I suspect the godot website used to encode those differently.
  • minor text edits that have been committed to upstream docs in the meantime
  • minor markup changes in the upstream HTML over time (mainly on the 3.x series)

Godot 4.2 was released on Nov 2023.

Since v3.5, godot upstream docs have used some new HTML layouts and file
structures. Since older versions still need to be scrapable, this
introduces another set of versioned filters for godot.

This change also makes small changes to the filters to handle the
current website markup for the previous godot doc versions.

The @GDscript and @globalscope entries can't currently be browsed, because of
an encoding mismatch between the frontend and backend; I've identified a
possible fix for that but will PR that separately.
@jessevanherk jessevanherk requested a review from a team as a code owner April 22, 2024 05:12
@gordon-boom
Copy link

Hi. Thanks for doing this!

I just tried your changes and getting a 403 when trying to download Godot 4.2 docs.

root@990bc51c4450:/devdocs# bundle exec thor docs:download godot@4.2
(1/1) Godot 4.2 FAILED (OpenURI::HTTPError: 403 Forbidden)
Done

I tried visiting https://docs.godotengine.org/en/4.2 on a browser and it doesn't look like I've been blacklisted or anything.

Regards

@jessevanherk
Copy link
Author

root@990bc51c4450:/devdocs# bundle exec thor docs:download godot@4.2

@gordon-boom I haven't done anything around uploading a packaged version to devdocs yet, so I wouldn't expect that to work at this point. If there's a step I need to do as part of this PR to get that working, let me know what it is and I'd be happy to do it!

The command I've been using thus far for testing is:
bundle exec thor docs:generate godot@4.2

@gordon-boom
Copy link

root@990bc51c4450:/devdocs# bundle exec thor docs:download godot@4.2

@gordon-boom I haven't done anything around uploading a packaged version to devdocs yet, so I wouldn't expect that to work at this point. If there's a step I need to do as part of this PR to get that working, let me know what it is and I'd be happy to do it!

The command I've been using thus far for testing is: bundle exec thor docs:generate godot@4.2

Ah silly me. I guess I didn't read the difference between download and generate. I will try that next time.

@gordon-boom
Copy link

Just tried it out. Looks good! Let's hope it gets merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants