Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add os-release file parsing, partially fix #25897 #33817

Open
wants to merge 1 commit into
base: devel
Choose a base branch
from

Conversation

mscherer
Copy link
Contributor

SUMMARY

Add os-release file parsing, partially fix #25897
/etc/os_release is documented on https://www.freedesktop.org/software/systemd/man/os-release.html

ISSUE TYPE
  • Feature Pull Request
COMPONENT NAME

setup

ANSIBLE VERSION
$ ansible --version
ansible 2.5.0 (add_fact_os_release 1b2081dfbb) last updated 2017/12/12 15:56:51 (GMT +200)
  config file = /home/misc/.ansible.cfg
  configured module search path = [u'/home/misc/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/misc/checkout/git/ansible/lib/ansible
  executable location = /home/misc/checkout/git/ansible/bin/ansible
  python version = 2.7.5 (default, May  3 2017, 07:55:04) [GCC 4.8.5 20150623 (Red Hat 4.8.5-14)]

@mscherer
Copy link
Contributor Author

I didn't took a look at testing that yet, but plan to add it to the unit tests.

@ansibot ansibot added affects_2.5 This issue/PR affects Ansible v2.5 feature_pull_request needs_triage Needs a first human triage before being processed. support:core This issue/PR relates to code supported by the Ansible Engineering Team. labels Dec 12, 2017
@ansibot ansibot added the test This PR relates to tests. label Dec 12, 2017
@mscherer mscherer force-pushed the add_fact_os_release branch 5 times, most recently from 401b9b0 to 0263852 Compare December 12, 2017 17:31
@ansibot
Copy link
Contributor

ansibot commented Dec 12, 2017

The test ansible-test sanity --test pep8 [?] failed with the following errors:

lib/ansible/module_utils/facts/system/os_release.py:37:19: E231 missing whitespace after ','
lib/ansible/module_utils/facts/system/os_release.py:51:1: W391 blank line at end of file

The test ansible-test sanity --test pylint [?] failed with the following error:

lib/ansible/module_utils/facts/system/os_release.py:51:0: trailing-newlines Trailing newlines

click here for bot help

@ansibot ansibot added the needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. label Dec 12, 2017
@ansibot
Copy link
Contributor

ansibot commented Dec 12, 2017

The test ansible-test sanity --test pep8 [?] failed with the following errors:

lib/ansible/module_utils/facts/system/os_release.py:37:19: E231 missing whitespace after ','
lib/ansible/module_utils/facts/system/os_release.py:51:1: W391 blank line at end of file

The test ansible-test sanity --test pylint [?] failed with the following error:

lib/ansible/module_utils/facts/system/os_release.py:51:0: trailing-newlines Trailing newlines

click here for bot help

@mscherer mscherer force-pushed the add_fact_os_release branch 3 times, most recently from f33b63e to de2c4d5 Compare December 13, 2017 10:36
@ansibot ansibot removed the needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. label Dec 13, 2017

def collect(self, module=None, collected_facts=None):
facts_dict = {}
path = None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A comment about The file /etc/os-release takes precedence over /usr/lib/os-release. could be added there.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

fact_namespace = 'ansible_os_release'
collector_class = OSReleaseFactCollector

@patch('ansible.module_utils.facts.system.os_release.get_file_content', return_value='# test\nID=clown_linux')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another variable with value using special characters outside of A–Z, a–z, 0–9 and enclosed with quotes might be added there.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added quotes, not sure about special char.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A space is good enough :)

@pilou-
Copy link
Contributor

pilou- commented Dec 13, 2017

Although it doesn't count: shipit .

@samdoran samdoran self-assigned this Dec 13, 2017
@gundalow gundalow removed the needs_triage Needs a first human triage before being processed. label Dec 13, 2017
Copy link
Contributor

@alikins alikins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to see the parse function be pulled out of the class. And for it to take a string for input and tests updated to use it.


def parse_file(self, path):
result = {}
for line in get_file_content(path).splitlines():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would split this method into:

  • reading in the file content to a string
  • parsing the string

Then the test cases can test the parsing without having to patch get_file_content or any file io.

May be easier to test if the parsing function was module scope method as well, then the parsing tests dont
need to construct a collector or a faux AnsibleModule. The OsReleaseFactCollector would still be useful for
adding tests for existence of /etc/os-release and general class tests.

for line in get_file_content(path).splitlines():
if line.startswith('#'):
continue
for t in shlex.split(line):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One odd thing that has come up with the os-release parsing code in module_utils/facts/system/distribution.py is the quoting of some of the right hand side of lines.

I'll track down the bug momentarily, but at least one distro did odd things about what was/wasn't quoted (either VERSION wasnt quoted or ID was quoted?). The lsb/os-release spec is kind of vague on that, and different '/bin/lsb_release' tools seemed to do different things.

This isn't something that has to be addressed since I don't think there was a clear correct answer, but just mentioning it for context.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, while lsb is vague on that, I think, but the os-release specs is clear:
"Variable assignment values must be enclosed in double or single quotes if they include spaces, semicolons or other special characters outside of A–Z, a–z, 0–9. Shell special characters ("$", quotes, backslash, backtick) must be escaped with backslashes, following shell style."

Now of course, being on the right side of the specs do not help users :/

fact_namespace = 'ansible_os_release'
collector_class = OSReleaseFactCollector

@patch('ansible.module_utils.facts.system.os_release.get_file_content', return_value='# test\nID=clown_linux\nVERSION="Clown Linux"')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to see more tests of the parsing, ideally with some examples of real os-release files. test/units/module_utils/test_distribtution.py[1] has some.

[1] test_distribution.py file is in the wrong place but nonetheless

@ansibot ansibot added the needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. label Dec 13, 2017
@alikins
Copy link
Contributor

alikins commented Dec 13, 2017

@mscherer #25725 and #31143 are the issue/pr related to the quoting in /etc/os-release I mentioned

@ansibot ansibot added feature This issue/PR relates to a feature request. and removed stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. stale_review Updates were made after the last review and the last review is more than 7 days old. feature_pull_request labels Feb 26, 2018
@ansibot ansibot added stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. stale_review Updates were made after the last review and the last review is more than 7 days old. labels Mar 10, 2018
@ansibot ansibot added the support:community This issue/PR relates to code supported by the Ansible community. label Sep 20, 2018
@ansibot ansibot removed the support:community This issue/PR relates to code supported by the Ansible community. label Nov 26, 2018
@bcoca bcoca added P3 Priority 3 - Approved, No Time Limitation needs_verified This issue needs to be verified/reproduced by maintainer labels Jan 18, 2019
@ansibot ansibot removed the stale_review Updates were made after the last review and the last review is more than 7 days old. label Jun 27, 2019
@ansibot ansibot added the stale_review Updates were made after the last review and the last review is more than 7 days old. label Jul 6, 2019
@ansibot ansibot added core_review In order to be merged, this PR must follow the core review workflow. and removed needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. stale_review Updates were made after the last review and the last review is more than 7 days old. labels Mar 3, 2020
@ansibot ansibot added needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. stale_review Updates were made after the last review and the last review is more than 7 days old. and removed core_review In order to be merged, this PR must follow the core review workflow. labels Mar 28, 2020
@ansibot ansibot added pre_azp This PR was last tested before migration to Azure Pipelines. and removed stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. stale_review Updates were made after the last review and the last review is more than 7 days old. labels Dec 6, 2020
@samdoran samdoran removed their assignment Jan 7, 2021
@ansibot ansibot added the needs_rebase https://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html label Mar 10, 2022
@ansibot ansibot added the stale_review Updates were made after the last review and the last review is more than 7 days old. label Jul 12, 2023
@ansibot ansibot added needs_rebase https://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html and removed needs_rebase https://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html labels Oct 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects_2.5 This issue/PR affects Ansible v2.5 feature This issue/PR relates to a feature request. has_issue needs_rebase https://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. needs_verified This issue needs to be verified/reproduced by maintainer P3 Priority 3 - Approved, No Time Limitation pre_azp This PR was last tested before migration to Azure Pipelines. stale_review Updates were made after the last review and the last review is more than 7 days old. support:core This issue/PR relates to code supported by the Ansible Engineering Team. test This PR relates to tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Atomic OS detection in Ansible Facts
8 participants