New grains fetching detailed hypervisor info for Xen domains #46459

ixs · 2018-03-09T10:01:26Z

What does this PR do?

Add new grains to parse hypervisor details

New Behavior

Xen exposes a lot of detail about the hypervisor in /sys/hypervisor.

Parse the hypervisor version and the capabilities and expose these
as grains.

This allows for detailed targeting and appropriate actions, e.g. when
running on XenServer install the XenServer guest-tools.

Sample output:

virtual_hv_features:
    000020f0
virtual_hv_features_list:
    - pae_pgdir_above_4gb
    - mmu_pt_update_preserve_ad
    - gnttab_map_avail_bits
    - memory_op_vnode_supported
virtual_hv_version:
    4.6.6-2.el7
virtual_hv_version_info:
    - 4
    - 6
    - .6-2.el7

Tests written?

No. It looked like the whole _virtual part has no tests at all.

Commits signed with GPG?

No

damon-atkins · 2018-03-09T12:19:40Z

Should these grains be documented? Is there any page/doco describing grains?

ixs · 2018-03-09T22:22:27Z

@damon-atkins I do not believe there is any documentation.

If you look at https://docs.saltstack.com/en/latest/topics/grains/, it links straight to https://github.com/saltstack/salt/blob/develop/salt/grains/core.py for "documentation". And otherwise it suggests to use the grains module to discover what is available.

rallytime · 2018-03-12T18:51:10Z

@ixs Can you clean up the lint violations? https://jenkins.saltstack.com/job/PR/job/salt-pr-lint-n/19996/violations/file/salt/grains/core.py/

ixs · 2018-03-14T09:19:21Z

@rallytime done.

rallytime · 2018-03-15T15:32:03Z

@ixs Looks like there are still some errors. You can run the lint check yourself if you like by following these instructions.

rallytime

I have a couple of requested changes here, too.

rallytime · 2018-03-15T15:32:45Z

salt/grains/core.py

+    try:
+        version = {}
+        for file in ('major', 'minor', 'extra'):
+            with salt.utils.fopen('/sys/hypervisor/version/{}'.format(file), 'r') as fhr:


Can you use the salt.utils.files.fopen path here, instead of the salt.utils.fopen? We've split out the utils init file into separate files in 2018.3 and develop.

Right. In that case you might also want to update your linter messages as that referes to salt.utils.fopen()

rallytime · 2018-03-15T15:33:21Z

salt/grains/core.py

+    # Try to get the exact hypervisor version from sysfs
+    try:
+        version = {}
+        for file in ('major', 'minor', 'extra'):


file is a reserved keyword in Python. Can you change this to another variable name?

rallytime · 2018-03-15T15:33:52Z

salt/grains/core.py

+                        13: 'memory_op_vnode_supported',
+                        14: 'ARM_SMCCC_supported'}
+    try:
+        with salt.utils.fopen('/sys/hypervisor/properties/features', 'r') as fhr:


Same comment here about the utils path. :)

Xen exposes a lot of detail about the hypervisor in /sys/hypervisor. Parse the hypervisor version and the capabilities and expose these as grains. This allows for detailed targeting and appropriate actions, e.g. when running on XenServer install the XenServer guest-tools. Sample output: virtual_hv_features: 000020f0 virtual_hv_features_list: - pae_pgdir_above_4gb - mmu_pt_update_preserve_ad - gnttab_map_avail_bits - memory_op_vnode_supported virtual_hv_version: 4.6.6-2.el7 virtual_hv_version_info: - 4 - 6 - .6-2.el7

terminalmage · 2018-03-20T19:19:51Z

salt/grains/core.py

+                version[fn] = fhr.read().strip()
+        grains['virtual_hv_version'] = '{}.{}{}'.format(version['major'], version['minor'], version['extra'])
+        grains['virtual_hv_version_info'] = (version['major'], version['minor'], version['extra'])
+    except:


Bare excepts are bad, at minimum except Exception should be used. But when you know the potenial exceptions, you should use them. In this case, except (IOError, OSError, KeyError):

terminalmage · 2018-03-20T19:26:11Z

salt/grains/core.py

+            with salt.utils.files.fopen('/sys/hypervisor/version/{}'.format(fn), 'r') as fhr:
+                version[fn] = fhr.read().strip()
+        grains['virtual_hv_version'] = '{}.{}{}'.format(version['major'], version['minor'], version['extra'])
+        grains['virtual_hv_version_info'] = (version['major'], version['minor'], version['extra'])


We should be using a list instead of a tuple here, for a couple reasons:

The tuple will be lost (converted to list) anyway if/when the data is serialized using msgpack, json, etc.

We allow grains to be added via YAML, which does not support tuples.

terminalmage · 2018-03-20T19:27:37Z

salt/grains/core.py

+        version = {}
+        for fn in ('major', 'minor', 'extra'):
+            with salt.utils.files.fopen('/sys/hypervisor/version/{}'.format(fn), 'r') as fhr:
+                version[fn] = fhr.read().strip()


This needs to be decoded to unicode using:

version[fn] = salt.utils.stringutils.to_unicode(fhr.read().strip())

This may also require that salt.utils.stringutils is added to the imports.

terminalmage · 2018-03-20T19:28:06Z

salt/grains/core.py

+                        14: 'ARM_SMCCC_supported'}
+    try:
+        with salt.utils.files.fopen('/sys/hypervisor/properties/features', 'r') as fhr:
+            features = fhr.read().strip()


Same as above with decoding to unicode.

terminalmage · 2018-03-20T19:29:24Z

salt/grains/core.py

+        with salt.utils.files.fopen('/sys/hypervisor/properties/features', 'r') as fhr:
+            features = fhr.read().strip()
+        enabled_features = []
+        for bit, feat in xen_feature_table.items():


Instead of xen_feature_table.items(), please use six.iteritems(xen_feature_table). This ensures that Python 2 uses a generator over the key/value pairs.

terminalmage · 2018-03-20T19:30:17Z

salt/grains/core.py

+                enabled_features.append(feat)
+        grains['virtual_hv_features'] = features
+        grains['virtual_hv_features_list'] = enabled_features
+    except:


Same as above with the bare exception. Try to narrow this down to the exceptions you'd expect to be raised.

cachedout · 2018-04-04T19:25:21Z

@ixs Are you planning to implement the review feedback on this?

cachedout · 2018-04-09T20:17:35Z

@ixs We did not hear back from you on the review feedback so we are going to close this. When you're ready to come back to it, please leave a comment here and we will happily re-open it. Thanks!

ixs · 2018-04-16T08:43:34Z

@cachedout thanks for the review and the suggestions. Finally gotten around to finish this. Please reopen.

rallytime · 2018-04-17T13:03:15Z

@ixs I have re-opened this. Looking forward to seeing your updates. :)

ixs · 2018-04-17T14:00:28Z

@rallytime Thanks. The new update is already up on github and I believe this PR has the latest changes with all requested work done.

rallytime · 2018-04-17T21:33:28Z

Hi @ixs - This change is causing some test failures in the test_core.py file. Can you take a look?

https://jenkins.saltstack.com/job/PR/job/salt-pr-linode-cent7-py3/4121/#showFailuresLink

This would happen if someone runs on bare metal, an admittedly increasingly rare situation...

ixs · 2018-04-18T08:04:06Z

@rallytime Ohh, that's new. Fixed.
Turns out, the virtual grain is of course not set when running not in a virtual environment... But who would ever do such a thing, right?

rallytime requested a review from garethgreenaway March 12, 2018 14:49

garethgreenaway approved these changes Mar 12, 2018

View reviewed changes

rallytime suggested changes Mar 15, 2018

View reviewed changes

ixs and others added 3 commits March 17, 2018 13:08

Incorporate lint suggested fixes.

7ca8109

Further fixes.

2d831fc

ixs requested a review from a team as a code owner March 17, 2018 12:11

ghost self-requested a review March 17, 2018 12:11

Fix superflous newline.

ee68215

terminalmage suggested changes Mar 20, 2018

View reviewed changes

cachedout closed this Apr 9, 2018

Incorporate code improvement suggestions from Mike

fd115c2

rallytime reopened this Apr 17, 2018

rallytime approved these changes Apr 17, 2018

View reviewed changes

rallytime requested a review from terminalmage April 17, 2018 14:29

terminalmage approved these changes Apr 17, 2018

View reviewed changes

Handle the case the osdata dict doesn't contain any virtual key.

095ca4f

This would happen if someone runs on bare metal, an admittedly increasingly rare situation...

Merge branch 'develop' into hypervisor_version

ef5e0d1

Merge branch 'develop' into hypervisor_version

d6e75fe

rallytime merged commit 5e13488 into saltstack:develop May 29, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New grains fetching detailed hypervisor info for Xen domains #46459

New grains fetching detailed hypervisor info for Xen domains #46459

ixs commented Mar 9, 2018

damon-atkins commented Mar 9, 2018

ixs commented Mar 9, 2018

rallytime commented Mar 12, 2018

ixs commented Mar 14, 2018

rallytime commented Mar 15, 2018

rallytime left a comment

rallytime Mar 15, 2018

ixs Mar 17, 2018

rallytime Mar 15, 2018

ixs Mar 17, 2018

rallytime Mar 15, 2018

ixs Mar 17, 2018

terminalmage Mar 20, 2018

terminalmage Mar 20, 2018

terminalmage Mar 20, 2018

terminalmage Mar 20, 2018

terminalmage Mar 20, 2018

terminalmage Mar 20, 2018

cachedout commented Apr 4, 2018

cachedout commented Apr 9, 2018

ixs commented Apr 16, 2018

rallytime commented Apr 17, 2018 •

edited

Loading

ixs commented Apr 17, 2018

rallytime commented Apr 17, 2018

ixs commented Apr 18, 2018

New grains fetching detailed hypervisor info for Xen domains #46459

New grains fetching detailed hypervisor info for Xen domains #46459

Conversation

ixs commented Mar 9, 2018

What does this PR do?

New Behavior

Tests written?

Commits signed with GPG?

damon-atkins commented Mar 9, 2018

ixs commented Mar 9, 2018

rallytime commented Mar 12, 2018

ixs commented Mar 14, 2018

rallytime commented Mar 15, 2018

rallytime left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cachedout commented Apr 4, 2018

cachedout commented Apr 9, 2018

ixs commented Apr 16, 2018

rallytime commented Apr 17, 2018 • edited Loading

ixs commented Apr 17, 2018

rallytime commented Apr 17, 2018

ixs commented Apr 18, 2018

rallytime commented Apr 17, 2018 •

edited

Loading