Fixes, more tests and some cleanup for CIM-XML tuple parser

Details, potentially visible to pywbem users:

- Fixed the issue that the VALUETYPE attribute of the KEYVALUE element was
  required in parse_keyvalue(). DSP0201 defines it as optional, with a default
  of 'string'. That is now implemented.

- If both the TYPE and VALUETYPE attributes are specified on a KEYVALUE element,
  TYPE now takes precedence. Previously, VALUETYPE took precedence. DSP0201
  leaves the precedence open in case of such discrepancies, but it seems more
  logical to let the more precise value take precedence.

- Fixed the issue that hex representations of integers in key values returned
  from the server were not supported.

- Fixed the issue that the tupleparser accepted characters for char16 typed
  values outside of the range for UCS-2 characters. Such characters are now
  rejected by raising ParseError.

- Fixed the issue that the tupleparser tolerated invalid child elements under
  a INSTANCE and ERROR elements. This now results in a ParseError being raised.

- Fixed the issue that the tupleparser did not default the `propagated` attribute
  to `False` (as required by DSP0201) when creating `CIMProperty` objects from
  the result of operations returning properties (e.g. GetInstance or GetClass).
  It does now.

- Improved the message text in `ParseError` exceptions raised by the tupleparser.

Details, not visible to pywbem users:

- Cleaned up the fact that the tupleparser did not default the `propagated`
  attribute to `False` (as required by DSP0201) when creating `CIMMethod`
  objects from the result of operations returning methods (e.g. GetClass).
  The parser now explicitly sets the `propagated` attribute to `False` if
  it is not specified in the CIM-XML. However, this is only a cleanup change,
  because the default value of the corresponding `CIMMethod` init parameter
  was already `False`, and so this change does not result in a different
  attribute value.

- As a consequence of the defaulting of propagated on properties, changed many
  test cases in the testclient yaml files to set propagated: false explicitly,
  on the expected objects.

- Added an assertion to pcdata() that the returned data is a unicode string type.
  This is guaranteed by the fact that the SAX parser uses the expat reader, even
  in case the string offered to its parseString() method is a byte string, but
  this is not obvious, and we want to know when it is not a unicode string.

- Changed the literal strings used for `join()` to be unicode literals, so
  the empty string produced when the join list is empty is also a unicode string.

- Removed the use of byname() in the tupleparser because in all cases, the lists
  of CIM objects can now be passed on directly.

- The byname() function in cim_obj.py now returns an OrderedDict that preserves
  the order of items in the input list. In addition, because that function
  is no longer used in pywbem, it now issues a DeprecationWarning (it was
  always in pywbem and could illegally have been used by users).

- Cleaned up calls to check_node() by removing any optional arguments that were
  empty lists, when the default value None had the same effect. Updated the
  description of check_node() to clarify how None is treated for the attribute
  lists.

- Now the test_client program preserves the order of yaml dicts when loading
  testcases from the yaml files. This required establishing the same
  order in the testcases between the pywbem API definitions and the HTTP
  messages. This is done using the yamlordereddictloader Pyton package. Added
  that package to the development dependencies.

- The `NAME` attribute for a PROPERTY element was listed in both
  the required and optional attribute list in its parsing function. Removing
  it from the optional list is a cleanup that is not externally visible.

- In wbem_request(), fixed the (internal) documentation about the returned
  type: It returns a byte string, not a unicode string.

- In the parse_...() functions, clarified the description in many (not all)
  cases.

- Moved all unpacking of numeric values into a new unpack_numeric() function.

- Moved all unpacking of string, char16 and datetime values into a new
  unpack_string() function. Strings are now ensured to be unicode.
  Datetime values are now returned as CIMDateTime objects.

- Added many testcases for the tupleparse parse_XXX() functions.

- Changed some variable names in `parse_...()` functions to be more consistent.

Signed-off-by: Andreas Maier <maiera@de.ibm.com>

dev-requirements.txt

@@ -18,6 +18,7 @@ httpretty>=0.8.14
 lxml>=4.0.0
 requests>=2.12.4
 decorator>=4.0.11
+yamlordereddictloader>=0.4.0
 
 # Unit test (no imports, invoked via py.test script):
 pytest-cov>=2.4.0

docs/changes.rst

@@ -133,6 +133,15 @@ Incompatible changes
   If you used this method and relied on the defaults being generated, you will
   now have to set these attributes explicitly.
 
+* If a WBEM server specifies contradicting TYPE and VALUETYPE attributes on a
+  KEYVALUE element returned to the client (this element is used in instance
+  paths, e.g. for the result of the EnumerateInstanceNames operation), TYPE now
+  takes precedence. Previously, VALUETYPE took precedence. DSP0201 leaves the
+  handling of such discrepancies open, and it seems more logical to let the
+  more precise value take precedence. Because WBEM servers are required to
+  specify consistent values for these attributes, this change should not affect
+  users of pywbem.
+
 Deprecations
 ^^^^^^^^^^^^
 
@@ -301,6 +310,9 @@ Enhancements
 * Docs: Clarified that CIM-XML multi-requests are not supported by pywbem
   and why that is not a functional limitation.
 
+* Improved the messages in ParseError exceptions raised when parsing CIM-XML
+  received from a WBEM server.
+
 Bug fixes
 ^^^^^^^^^
 
@@ -425,6 +437,27 @@ Bug fixes
   characters on namespace names are stripped off in pywbem before sending
   the request to the server. (Issue #255).
 
+* Fixed the issue that the parser for CIM-XML received from the WBEM server
+  required the VALUETYPE attribute of the KEYVALUE element. DSP0201 defines
+  VALUETYPE as optional, with a default of 'string'. That is now implemented.
+
+* Fixed the issue that the parser for CIM-XML received from the WBEM server
+  did not support hexadecimal representations of integers in the KEYVALUE
+  element. They are now supported.
+
+* Fixed the issue that the parser for CIM-XML received from the WBEM server
+  accepted characters for char16 typed values outside of the range for
+  UCS-2 characters. Such characters are now rejected by raising ParseError.
+
+* Fixed the issue that the parser for CIM-XML received from the WBEM server
+  tolerated invalid child elements under INSTANCE or ERROR elements. This
+  now results in a ParseError being raised.
+
+* Fixed the issue that the parser for CIM-XML received from the WBEM server
+  did not set the `propagated` attribute to False in `CIMProperty` objects
+  retrieved from operations (e.g. as part of a class or instance), as
+  required by DSP0201. It does now.
+
 Cleanup
 ^^^^^^^
 

minimum-constraints.txt

@@ -70,6 +70,7 @@ httpretty===0.8.14
 lxml===4.0.0
 requests===2.12.4
 decorator===4.0.11
+yamlordereddictloader===0.4.0
 
 # Unit test (no imports, invoked via py.test script):
 pytest-cov===2.4.0

pywbem/cim_http.py

@@ -429,7 +429,7 @@ def wbem_request(url, data, creds, cimxml_headers=None, debug=False, x509=None,
         Tuple containing:
 
             The CIM-XML formatted response data from the WBEM server, as a
-            :term:`unicode string` object.
+            :term:`byte string` object.
 
             The server response time in seconds as floating point number if
             this data was received from the server. If no data returned

pywbem/cim_obj.py

@@ -7783,6 +7783,12 @@ def _check_embedded_object(embedded_object, type, value, element_txt):
 
 def byname(nlist):
     """
-    Convert a list of named objects into a map indexed by name
+    Convert a list of named objects into an ordered dictionary indexed by name.
+
+    Deprecated: This function is internal and has been deprecated in pywbem
+    0.12.
     """
-    return dict([(x.name, x) for x in nlist])
+    warnings.warn("The internal byname() function has been deprecated, with "
+                  "no replacement.", DeprecationWarning,
+                  stacklevel=_stacklevel_above_module(__name__))
+    return OrderedDict([(x.name, x) for x in nlist])