Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add os-release file parsing, partially fix #25897 #33817

Open
wants to merge 1 commit into
base: devel
from

Conversation

@mscherer
Copy link
Contributor

@mscherer mscherer commented Dec 12, 2017

SUMMARY

Add os-release file parsing, partially fix #25897
/etc/os_release is documented on https://www.freedesktop.org/software/systemd/man/os-release.html

ISSUE TYPE
  • Feature Pull Request
COMPONENT NAME

setup

ANSIBLE VERSION
$ ansible --version
ansible 2.5.0 (add_fact_os_release 1b2081dfbb) last updated 2017/12/12 15:56:51 (GMT +200)
  config file = /home/misc/.ansible.cfg
  configured module search path = [u'/home/misc/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/misc/checkout/git/ansible/lib/ansible
  executable location = /home/misc/checkout/git/ansible/bin/ansible
  python version = 2.7.5 (default, May  3 2017, 07:55:04) [GCC 4.8.5 20150623 (Red Hat 4.8.5-14)]
@mscherer
Copy link
Contributor Author

@mscherer mscherer commented Dec 12, 2017

I didn't took a look at testing that yet, but plan to add it to the unit tests.

@mscherer mscherer force-pushed the mscherer:add_fact_os_release branch Dec 12, 2017
@ansibot ansibot added the test label Dec 12, 2017
@mscherer mscherer force-pushed the mscherer:add_fact_os_release branch 5 times, most recently Dec 12, 2017
@ansibot
Copy link
Contributor

@ansibot ansibot commented Dec 12, 2017

The test ansible-test sanity --test pep8 [?] failed with the following errors:

lib/ansible/module_utils/facts/system/os_release.py:37:19: E231 missing whitespace after ','
lib/ansible/module_utils/facts/system/os_release.py:51:1: W391 blank line at end of file

The test ansible-test sanity --test pylint [?] failed with the following error:

lib/ansible/module_utils/facts/system/os_release.py:51:0: trailing-newlines Trailing newlines

click here for bot help

@mscherer mscherer force-pushed the mscherer:add_fact_os_release branch Dec 12, 2017
@ansibot
Copy link
Contributor

@ansibot ansibot commented Dec 12, 2017

The test ansible-test sanity --test pep8 [?] failed with the following errors:

lib/ansible/module_utils/facts/system/os_release.py:37:19: E231 missing whitespace after ','
lib/ansible/module_utils/facts/system/os_release.py:51:1: W391 blank line at end of file

The test ansible-test sanity --test pylint [?] failed with the following error:

lib/ansible/module_utils/facts/system/os_release.py:51:0: trailing-newlines Trailing newlines

click here for bot help

@mscherer mscherer force-pushed the mscherer:add_fact_os_release branch 3 times, most recently Dec 13, 2017
lib/ansible/module_utils/facts/system/os_release.py Outdated

def collect(self, module=None, collected_facts=None):
facts_dict = {}
path = None

This comment has been minimized.

@pilou-

pilou- Dec 13, 2017
Contributor

A comment about The file /etc/os-release takes precedence over /usr/lib/os-release. could be added there.

This comment has been minimized.

@mscherer

mscherer Dec 13, 2017
Author Contributor

Done

test/units/module_utils/facts/test_collectors.py Outdated
fact_namespace = 'ansible_os_release'
collector_class = OSReleaseFactCollector

@patch('ansible.module_utils.facts.system.os_release.get_file_content', return_value='# test\nID=clown_linux')

This comment has been minimized.

@pilou-

pilou- Dec 13, 2017
Contributor

Another variable with value using special characters outside of A–Z, a–z, 0–9 and enclosed with quotes might be added there.

This comment has been minimized.

@mscherer

mscherer Dec 13, 2017
Author Contributor

I added quotes, not sure about special char.

This comment has been minimized.

@pilou-

pilou- Dec 13, 2017
Contributor

A space is good enough :)

@mscherer mscherer force-pushed the mscherer:add_fact_os_release branch Dec 13, 2017
@pilou-
Copy link
Contributor

@pilou- pilou- commented Dec 13, 2017

Although it doesn't count: shipit .

Copy link
Contributor

@alikins alikins left a comment

I'd like to see the parse function be pulled out of the class. And for it to take a string for input and tests updated to use it.

lib/ansible/module_utils/facts/system/os_release.py Outdated

def parse_file(self, path):
result = {}
for line in get_file_content(path).splitlines():

This comment has been minimized.

@alikins

alikins Dec 13, 2017
Contributor

I would split this method into:

  • reading in the file content to a string
  • parsing the string

Then the test cases can test the parsing without having to patch get_file_content or any file io.

May be easier to test if the parsing function was module scope method as well, then the parsing tests dont
need to construct a collector or a faux AnsibleModule. The OsReleaseFactCollector would still be useful for
adding tests for existence of /etc/os-release and general class tests.

lib/ansible/module_utils/facts/system/os_release.py Outdated
for line in get_file_content(path).splitlines():
if line.startswith('#'):
continue
for t in shlex.split(line):

This comment has been minimized.

@alikins

alikins Dec 13, 2017
Contributor

One odd thing that has come up with the os-release parsing code in module_utils/facts/system/distribution.py is the quoting of some of the right hand side of lines.

I'll track down the bug momentarily, but at least one distro did odd things about what was/wasn't quoted (either VERSION wasnt quoted or ID was quoted?). The lsb/os-release spec is kind of vague on that, and different '/bin/lsb_release' tools seemed to do different things.

This isn't something that has to be addressed since I don't think there was a clear correct answer, but just mentioning it for context.

This comment has been minimized.

@mscherer

mscherer Dec 14, 2017
Author Contributor

So, while lsb is vague on that, I think, but the os-release specs is clear:
"Variable assignment values must be enclosed in double or single quotes if they include spaces, semicolons or other special characters outside of A–Z, a–z, 0–9. Shell special characters ("$", quotes, backslash, backtick) must be escaped with backslashes, following shell style."

Now of course, being on the right side of the specs do not help users :/

test/units/module_utils/facts/test_collectors.py Outdated
fact_namespace = 'ansible_os_release'
collector_class = OSReleaseFactCollector

@patch('ansible.module_utils.facts.system.os_release.get_file_content', return_value='# test\nID=clown_linux\nVERSION="Clown Linux"')

This comment has been minimized.

@alikins

alikins Dec 13, 2017
Contributor

I'd like to see more tests of the parsing, ideally with some examples of real os-release files. test/units/module_utils/test_distribtution.py[1] has some.

[1] test_distribution.py file is in the wrong place but nonetheless

@alikins
Copy link
Contributor

@alikins alikins commented Dec 13, 2017

@mscherer #25725 and #31143 are the issue/pr related to the quoting in /etc/os-release I mentioned

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

8 participants