Scraping changes needed before November 16th - AWS Documentation changes coming

[kmcquade]

I just got this from the IAM Documentation team (first time I learned there is such a team!)

---

We chatted a while back about your Policy Sentry code in GitHub that scrapes the IAM Actions, resources, and condition keys documentation. We will delete those pages in 30 days and redirect traffic to the matching page in the new Service Authorization Reference.

Action item: Within the next 30 days, we recommend that you review the policy_sentry code and update it to scrape the new pages. The content format of these new pages and the HTML file names have not changed. If your code supports server redirects, then it should not be affected at all. However, we still recommend that you use the new URL.

Today:

• The existing Actions, resources, and condition keys pages include a new “Tip” box at the top of the page. The link in that paragraph takes you to the same page in the new guide. https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_actions-resources-contextkeys.html
• The new guide includes the same structure, file names, and content as the IAM User Guide. However, the table of contents (TOC) structure of the guide is different above the page. The IAM User Guide content is located at Reference > Policy reference > Actions, resources and condition keys. The new Service Authorization Reference content is located in Reference > Actions, resources and condition keys.
• You can begin testing your code using the new URL: https://docs.aws.amazon.com/service-authorization/latest/reference/reference.html.

On November 16:

• The AWS Docs server will begin permanently redirecting customers from the old page to the matching page in the new guide.
• The IAM User Guide TOC will still include the high-level Actions, resources, and condition keys page. Choosing that TOC entry redirects you to the new guide.
• On or before that date, we will remove the existing Actions, resources, and condition keys pages from GitHub.

If you have any questions or recommendations for documentation improvements, please let us know.

Thanks,
IAM Documentation

---

FYI - @aidansteele, @0xdabbad00

[kmcquade]

Update: To my surprise, the scraping process still worked by just changing the URL.

policy_sentry/policy_sentry/shared/constants.py

Line 22 in e6d275b

BASE_DOCUMENTATION_URL = "https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_actions-resources-contextkeys.html"

Let's assume that will work for now, but might break at some point on or before November 16th. In the meantime, I'll leave this issue open.

[reetasingh]

@kmcquade are we on track for this?

[0xdabbad00]

The change just went through last night. Parliament's scraping is broken. I'll look into it on my end. awsdocs/iam-user-guide@0eb32aa

[0xdabbad00]

All I had to do for Parliament was specify different URLs: duo-labs/parliament@d371b6b

My scraping works now.

[kmcquade]

I believe this was fixed by @saikirankv in #258

I'll wait to close this until the automation that updates the documentation runs. Once that PR happens and we can verify it was successful, I will close.

[kmcquade]

Fixed by #307