Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

no-index index.html pages that are redirects. #5

Open
glenrobson opened this issue Mar 9, 2022 · 3 comments · Fixed by #6
Open

no-index index.html pages that are redirects. #5

glenrobson opened this issue Mar 9, 2022 · 3 comments · Fixed by #6

Comments

@glenrobson
Copy link
Member

E.g.:

https://iiif.io/api/image/2/index.html
User-declared canonical: https://iiif.io/api/image/2.1/

Issue sitemap shows the index.html as that is what is in s3 but the HTML does a redirect (including noindex) to the directory name without the index.html:

<!DOCTYPE html>
<html lang="en-US">
  <meta charset="utf-8">
  <title>Redirecting&hellip;</title>
  <link rel="canonical" href="/api/image/2.1/">
  <script>location="/api/image/2.1/"</script>
  <meta http-equiv="refresh" content="0; url=/api/image/2.1/">
  <meta name="robots" content="noindex">
  <h1>Redirecting&hellip;</h1>
  <a href="/api/image/2.1/">Click here if you are not redirected.</a>
</html>

This is due to the source file having a redirect_from: https://github.com/IIIF/api/blob/main/source/image/2.1/index.md. Same fix as: IIIF/website#616
@glenrobson
Copy link
Member Author

Will fix:

https://iiif.io/api/image/3/index.html
https://iiif.io/api/model/shared-canvas/index.html
https://iiif.io/api/presentation/3/index.html
https://iiif.io/api/image/3/index.html
https://iiif.io/api/discovery/index.html
https://iiif.io/api/discovery/0/index.html
https://iiif.io/api/search/1/index.html
https://iiif.io/api/model/index.html
https://iiif.io/api/discovery/1/index.html

and probably a few more.

@glenrobson glenrobson transferred this issue from IIIF/website Mar 11, 2022
@glenrobson glenrobson mentioned this issue Mar 19, 2022
Merged
@glenrobson
Copy link
Member Author

Seems to be failing at parsing the html here:



loading 404.html
--
loading about.html
loading about/index.html
Found duplicate: https://iiif.io//get-started/why-iiif/ from https://iiif.io/about/index.html
loading api/0/content-state/index.html
loading api/annex/index.html
loading api/annex/notes/apache.html
loading api/annex/notes/apache/index.html
loading api/annex/notes/design_patterns/index.html
loading api/annex/notes/design_principles/index.html
loading api/annex/notes/disclaimer.html
loading api/annex/notes/disclaimer/index.html
loading api/annex/notes/index.html
loading api/annex/notes/jsonld.html
loading api/annex/notes/jsonld/index.html
loading api/annex/notes/rotation.html
loading api/annex/notes/rotation/index.html
loading api/annex/notes/semver.html
loading api/annex/notes/semver/index.html
loading api/annex/openannotation/index.html
loading api/annex/rest/index.html
loading api/annex/services/index.html
loading api/auth/0.9/implementation.html
loading api/auth/0.9/implementation/index.html
loading api/auth/0.9/index.html
loading api/auth/0/clickthrough/index.html
loading api/auth/0/external/index.html
loading api/auth/0/index.html
loading api/auth/0/kiosk/index.html
loading api/auth/0/login/index.html
loading api/auth/0/token/index.html
loading api/auth/1.0/implementation.html
loading api/auth/1.0/implementation/index.html
loading api/auth/1.0/index.html
loading api/auth/1/clickthrough/index.html
loading api/auth/1/external/index.html
loading api/auth/1/index.html
loading api/auth/1/kiosk/index.html
loading api/auth/1/login/index.html
loading api/auth/1/token/index.html
loading api/auth/index.html
Found duplicate: https://iiif.io//api/auth/1.0/ from https://iiif.io/api/auth/index.html
loading api/content-state/0.1/index.html
loading api/content-state/0.2/index.html
loading api/content-state/0.3/index.html
loading api/content-state/0.9/index.html
loading api/content-state/0/index.html
loading api/content-state/1.0/index.html
loading api/content-state/1/index.html
loading api/content-state/index.html
Found duplicate: https://iiif.io//api/content-state/1.0/ from https://iiif.io/api/content-state/index.html
loading api/cookbook/index.html
loading api/cookbook/recipe/0000_template/index.html
loading api/cookbook/recipe/0001-mvm-image/index.html
loading api/cookbook/recipe/0002-mvm-audio/index.html
loading api/cookbook/recipe/0003-mvm-video/index.html
loading api/cookbook/recipe/0004-canvas-size/index.html
loading api/cookbook/recipe/0005-image-service/index.html
loading api/cookbook/recipe/0006-text-language/index.html
loading api/cookbook/recipe/0007-string-formats/index.html
loading api/cookbook/recipe/0008-rights/index.html
loading api/cookbook/recipe/0009-book-1/index.html
loading api/cookbook/recipe/0010-book-2-viewing-direction/index.html
loading api/cookbook/recipe/0011-book-3-behavior/index.html
loading api/cookbook/recipe/0012-thumbnails/index.html
loading api/cookbook/recipe/0013-placeholderCanvas/index.html
loading api/cookbook/recipe/0014-accompanyingcanvas/index.html
loading api/cookbook/recipe/0015-start/index.html
loading api/cookbook/recipe/0016-transcription-image/index.html
loading api/cookbook/recipe/0017-transcription-av/index.html
loading api/cookbook/recipe/0018-transcription-xml/index.html
loading api/cookbook/recipe/0019-comments/index.html
loading api/cookbook/recipe/0020-fragment-selector/index.html
loading api/cookbook/recipe/0021-tagging/index.html
loading api/cookbook/recipe/0022-linking/index.html
loading api/cookbook/recipe/0023-annotating-specific-resources/index.html
loading api/cookbook/recipe/0024-book-4-toc/index.html
loading api/cookbook/recipe/0025-toc-newspaper-articles/index.html
loading api/cookbook/recipe/0026-toc-opera/index.html
loading api/cookbook/recipe/0027-alt-sequence/index.html
loading api/cookbook/recipe/0028-sequence-range-partial-canvases/index.html

So sitemap isn't updated.

@glenrobson glenrobson reopened this Mar 19, 2022
@glenrobson
Copy link
Member Author

I think these are fixed but it runs longer than 30s which causes a timeout on the AWS API gateway which means GitHub thinks it has failed.

@glenrobson glenrobson mentioned this issue Apr 4, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Adding code to read the HTML looking for redirects IIIF/s3-sitemap
1 participant
@glenrobson