BB-749 Update robots.txt.j2 to accept Allow rule and multiple values

[john2x]

Configuration Pull Request

This PR updates the robots.txt.j2 template to satisfy the following:

• accept Allow rules
• accept multiple values for Agent, Disallow, and Allow as a list of strings
• maintain backward compatibility of old var values (i.e. single string values)

Testing

To test the template, checkout this branch and add a test YAML file containing the NGINX_ROBOT_RULES values, and run ansible to render the template:

$ cd path/to/edx/configuration/configuration/playbooks/roles/nginx/templates/edx/app/nginx
$ vim test.yaml
...
$ ansible all -i localhost, -c local -m template -a "src=robots.txt.j2 dest=./test.txt" --extra-vars=@test.yaml
$ cat test.txt

This will render the template and save it to test.txt.

Here are some sample values for NGINX_ROBOT_RULES and their expected output:

# mix of single values and list of values
NGINX_ROBOT_RULES:
  - agent: '*'
    disallow: '/'
    allow:
      - '/$'
      - '/courses$'
      - '/courses/*/about$'
  - agent: 'W3C-checklink'  # backward compatible agent
    disallow: ''  # backward compatible disallow rule

# --- output ---
User-agent: *
Allow: /$
Allow: /courses$
Allow: /courses/*/about$
Disallow: /

User-agent: W3C-checklink
Disallow:

# singlue values only (for backward compatibility)
NGINX_ROBOT_RULES:
  - agent: '*'
    disallow: '/'
  - agent: 'W3C-checklink'
    disallow: ''

# --- output ---
User-agent: *
Disallow: /

User-agent: W3C-checklink
Disallow:

# all multiple values (lists of strings)
NGINX_ROBOT_RULES:
  - agent:
      - 'foo'
      - 'bar'
    disallow:
      - '/'
    allow:
      - '/$'
      - '/courses$'
      - '/courses/*/about$'
  - agent:
      - 'fizz'
    disallow:
      - '/'

# --- output ---
User-agent: foo
User-agent: bar
Allow: /$
Allow: /courses$
Allow: /courses/*/about$
Disallow: /

User-agent: fizz
Disallow: /

---

Discussions: https://gitlab.com/opencraft/client/lumerical/configuration/merge_requests/5

Reviewers

• @lgp171188
• edX reviewer[s] TBD

---

Make sure that the following steps are done before merging:

• A DevOps team member has approved the PR if it is code shared across multiple services and you don't own all of the services.
• Are you adding any new default values that need to be overridden when this change goes live? If so:
  • Update the appropriate internal repo (be sure to update for all our environments)
  • If you are updating a secure value rather than an internal one, file a DEVOPS ticket with details.
  • Add an entry to the CHANGELOG.
• If you are making a complicated change, have you performed the proper testing specified on the Ops Ansible Testing Checklist? Adding a new variable does not require the full list (although testing on a sandbox is a great idea to ensure it links with your downstream code changes).

[openedx-webhooks]

Thanks for the pull request, @john2x! I've created OSPR-2977 to keep track of it in JIRA. JIRA is a place for product owners to prioritize feature reviews by the engineering development teams.

Feel free to add as much of the following information to the ticket:

• supporting documentation
• edx-code email threads
• timeline information ("this must be merged by XX date", and why that is)
• partner information ("this is a course on edx.org")
• any other information that can help Product understand the context for the PR

All technical communication about the code itself will still be done via the GitHub pull request interface. As a reminder, our process documentation is here.

[natabene]

@john2x Thank you for your contribution, please let me know once it is ready to be looked at.

[john2x]

@natabene This is ready to be looked at, thanks.

[lgp171188]

👍

• I tested this by running the playbook and verifying the generated robots.txt file for various values.
• I read through the code
• I checked for accessibility issues NA
• Includes documentation NA

[lgp171188]

@john2x, can you update this PR to resolve the merge conflict?

[natabene]

@john2x Can you let me know once you resolve the merge conflict?

[john2x]

@natabene @lgp171188 I've rebased to the latest change from master and resolved the conflict.

[natabene]

@john2x Thanks! @edx/devops This is ready for your review, when you have time.

[john2x]

@natabene do you have updates for the review/prioritization of this PR?

[natabene]

@john2x Sorry, we have not had a chance to review it yet, I will ping team again. Do you know why checks are failing now?

[john2x]

@natabene Thanks! The checks was because of conflicts on the CHANGELOG.md file due to recent changes on master. I have resolved the latest one.

[natabene]

@john2x Thank you! @edx/devops Can you give this a look?

[jdmulloy]

Looks good to me.

[natabene]

@jdmulloy Can you merge, too? Open edX community contributors don't have the necessary permissions to merge.

[openedx-webhooks]

@john2x 🎉 Your pull request was merged!

Please take a moment to answer a two question survey so we can improve your experience in the future.

[lgp171188]

@natabene, can this be cherry-picked to Ironwood?

[nedbat]

I've cherry-picked this onto Ironwood.