Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Removing from a repo with mof_compiler script #1158

Closed
KSchopmeyer opened this issue Apr 1, 2018 · 10 comments
Closed

Removing from a repo with mof_compiler script #1158

KSchopmeyer opened this issue Apr 1, 2018 · 10 comments

Comments

@KSchopmeyer
Copy link
Collaborator

Removing items from the repo with mof_compiler appears to be incomplete. After a correct install,

We executed the following:


(pywbem27) kschopmeyer@sheldon:~/pywbem/githubpywbem/pywbem(master)$ mof_compiler -r -s http://sheldon testsuite/test.mof
Error deleting instance root/cimv2:PyWBEM_AllTypes
     6 CIM_ERR_NOT_FOUND: PyWBEM_AllTypes
Error deleting instance root/cimv2:PyWBEM_AllTypes
     6 CIM_ERR_NOT_FOUND: PyWBEM_AllTypes
Error deleting instance root/cimv2:PyWBEM_MemberOfPersonCollection
     6 CIM_ERR_NOT_FOUND: PyWBEM_MemberOfPersonCollection
Error deleting instance root/cimv2:PyWBEM_MemberOfPersonCollection
     6 CIM_ERR_NOT_FOUND: PyWBEM_MemberOfPersonCollection
Error deleting instance root/cimv2:PyWBEM_MemberOfPersonCollection
     6 CIM_ERR_NOT_FOUND: PyWBEM_MemberOfPersonCollection
Error deleting instance root/cimv2:PyWBEM_PersonCollection
     6 CIM_ERR_NOT_FOUND: PyWBEM_PersonCollection
Error deleting class root/cimv2:PyWBEM_AllTypes
     9 CIM_ERR_CLASS_HAS_INSTANCES: PyWBEM_AllTypes
Error deleting class root/cimv2:PyWBEM_MemberOfPersonCollection
     9 CIM_ERR_CLASS_HAS_INSTANCES: PyWBEM_MemberOfPersonCollection
Error deleting class root/cimv2:PyWBEM_PersonCollection
     9 CIM_ERR_CLASS_HAS_INSTANCES: PyWBEM_PersonCollection

The results are inconsistent.

  1. The PyWBEM_AllTypes class is definitily in the repo.

    cimcli gc PyWBEM_AllTypes

    // ===================================================
    // PyWBEM_AllTypes
   // ===================================================

       [Version ( "1.0.0" ), 

  1. Instances still exist for PyWBEM_MemberOfPersonCollection

kschopmeyer@sheldon:~/pgogtest$ cimcli ni PyWBEM_MemberOfPersonCollection
PyWBEM_MemberOfPersonCollection.Collection="PyWBEM_PersonCollection.InstanceID=\"PersonCollection\"",Member="PyWBEM_Person.CreationClassName=\"PyWBEM_Person\",Name=\"Alice\""
PyWBEM_MemberOfPersonCollection.Collection="PyWBEM_PersonCollection.InstanceID=\"PersonCollection\"",Member="PyWBEM_Person.CreationClassName=\"PyWBEM_Person\",Name=\"Bob\""
PyWBEM_MemberOfPersonCollection.Collection="PyWBEM_PersonCollection.InstanceID=\"PersonCollection\"",Member="PyWBEM_Person.CreationClassName=\"PyWBEM_Person\",Name=\"Charlie\""

  1. Any yet it does delete all of the members of PyWBEM_Person and the class.

cimcli ni PyWBEM_Person
cimcli CIMException:  Cmd= ni Object= PyWBEM_Person Code= 5
CIM_ERR_INVALID_CLASS: PyWBEM_Person

@andy-maier
Copy link
Contributor

Discussion point: Mandatory for 0.13.0?

@andy-maier
Copy link
Contributor

We decided in the call today that this would be considered optional for 0.13.0

@andy-maier
Copy link
Contributor

We decided to move this issue to the next release.

@andy-maier andy-maier modified the milestones: 0.13.0, 1.0.0 Oct 10, 2018
@andy-maier andy-maier added the roll back needed Roll back into latest fix branch is needed label Nov 2, 2019
@andy-maier
Copy link
Contributor

We decided to put this into 1.0.0 and roll it back to 0.15.0. Setting discussion needed to keep this one above the horizon.

@KSchopmeyer
Copy link
Collaborator Author

KSchopmeyer commented Nov 9, 2019

I know the problems now.

  1. The instance names we use to execute the remove do not match the instance names in the server. See First Issue below.

  2. For some association classes, the at least one reference property has a path with no keybindings. See Second Issue below.

First issue:

It is the compiler again and our use of it in mofcomp -r. The remove logic compiles the defined files into the local MOFWBEMConnection.

However, when we send the delete requests for the instancesto the server we do not have a complete path since we compiled them without aliases which is what sets the path component in the compiler. WHOOPS. So the server simply cannot find the instances to remove. Since the server cannot find and remove instances, it also fails any class removal because there are instances.

Proposed Fix: Create a new adaption of MOFWBEMConnection that adds path as the files to be removed are recompiled.

Here is an example of the request as Pegasus sees it:


1573325449s-589284us: XmlIO [3409:139696727299968:HTTPConnection.cpp:2275]: <!-- No request message received; connection closed: queue id: 23 -->
1573325451s-934021us: XmlIO [3409:139696727299968:HTTPConnection.cpp:2319]: <!-- Request: queue id: 23 -->
POST /cimom HTTP/1.1
Host: localhost:5988
Accept-Encoding: identity
Content-type: application/xml; charset="utf-8"
Content-length: 388
PegasusAuthorization: Local "kschopmeyer"
CIMOperation: MethodCall
CIMMethod: DeleteInstance
CIMObject: root/cimv2

<?xml version="1.0" encoding="utf-8" ?>
<CIM CIMVERSION="2.0" DTDVERSION="2.0"><MESSAGE ID="1001" PROTOCOLVERSION="1.0">
<SIMPLEREQ><IMETHODCALL NAME="DeleteInstance"><LOCALNAMESPACEPATH><NAMESPACE NAME="root"/><NAMESPACE NAME="cimv2"/></LOCALNAMESPACEPATH>
<IPARAMVALUE NAME="InstanceName"><INSTANCENAME CLASSNAME="PyWBEM_AllTypes"/></IPARAMVALUE></IMETHODCALL></SIMPLEREQ></MESSAGE></CIM>
1573325451s-936044us: XmlIO [3409:139696727299968:HTTPConnection.cpp:924]: <!-- Response: queue id: 23 -->
HTTP/1.1 200 OK
Content-Type: application/xml; charset=utf-8
content-length: 0000000290
CIMOperation: MethodResponse

<?xml version="1.0" encoding="utf-8" ?>
<CIM CIMVERSION="2.0" DTDVERSION="2.0">
<MESSAGE ID="1001" PROTOCOLVERSION="1.0">
<SIMPLERSP>
<IMETHODRESPONSE NAME="DeleteInstance">
<ERROR CODE="6" DESCRIPTION="CIM_ERR_NOT_FOUND: PyWBEM_AllTypes"/></IMETHODRESPONSE>
</SIMPLERSP>
</MESSAGE>
</CIM>

Second Issue:
When the mof is compiled into MOFWBEMConnection, The compliation is very simplistic so that it does not handle things like override, etc. The end result is that classes that are associations and subclasses of some other associaiton with a key override (ex. Subclasses of CIM_Collection that includes InstanceID but without the key qualifier, the key qualifier does not get set on the subclass. Then when we try to build the instance path from the instance to do the remove, it ends up with a NULL instancekeybindings which means that the instance path we send to the server to delete instance is not the same as the path in the server.

Issues Found:

a. Compiler not passing LocalOnly=False, IncludeQualifiers=True in some cases of GetClass (which is recursive) so that in the recursion, the class was returned with key qualifier not set or key properties not in class. This causes reference properties in Association classes to be compiled with empty keybindings.

b. In reference properties of Association classes where the value is defined by an instance alias, the CIMNamespace is defined with the namespace component set.

DISCUSSION
I am not sure what the correct answer to this is. I know OpenPegasus leaves the namespace component of the reference property value as None for compiled instances that use alias to define the reference property value. Since the pywbem MOF compiler was inserting the namespace, the result was instance paths that do not match if the reference properties were keys.

So if an association class where the reference properties are keys (Sadly, CIM_Collection and all it subclasses) is compiled from MOF by the pegasus MOF compiler but the pywbem mof compiler is used to remove the instances, pegasus cannot find the instances because each has the namespace set in the path of the Delete request but not in the path in the WBEM Server.

@KSchopmeyer
Copy link
Collaborator Author

Note that pegasus also returns NOT_FOUND for at least one class, Pywbem_Person but I have not sorted out the reason for that.

1573325451s-977797us: XmlIO [3409:139696727299968:HTTPConnection.cpp:2275]: <!-- No request message received; connection closed: queue id: 23 -->
1573325451s-979420us: XmlIO [3409:139696727299968:HTTPConnection.cpp:2319]: <!-- Request: queue id: 23 -->
POST /cimom HTTP/1.1
Host: localhost:5988
Accept-Encoding: identity
Content-type: application/xml; charset="utf-8"
Content-length: 372
PegasusAuthorization: Local "kschopmeyer"
CIMOperation: MethodCall
CIMMethod: DeleteClass
CIMObject: root/cimv2

<?xml version="1.0" encoding="utf-8" ?>
<CIM CIMVERSION="2.0" DTDVERSION="2.0"><MESSAGE ID="1001" PROTOCOLVERSION="1.0">
<SIMPLEREQ><IMETHODCALL NAME="DeleteClass"><LOCALNAMESPACEPATH><NAMESPACE NAME="root"/><NAMESPACE NAME="cimv2"/>
</LOCALNAMESPACEPATH><IPARAMVALUE NAME="ClassName">
<CLASSNAME NAME="PyWBEM_Person"/></IPARAMVALUE></IMETHODCALL></SIMPLEREQ></MESSAGE></CIM>
1573325451s-979762us: XmlIO [3409:139696727299968:HTTPConnection.cpp:924]: <!-- Response: queue id: 23 -->
HTTP/1.1 200 OK
Content-Type: application/xml; charset=utf-8
content-length: 0000000285
CIMOperation: MethodResponse

<?xml version="1.0" encoding="utf-8" ?>
<CIM CIMVERSION="2.0" DTDVERSION="2.0">
<MESSAGE ID="1001" PROTOCOLVERSION="1.0">
<SIMPLERSP>
<IMETHODRESPONSE NAME="DeleteClass">
<ERROR CODE="6" DESCRIPTION="CIM_ERR_NOT_FOUND: PyWBEM_Person"/></IMETHODRESPONSE>
</SIMPLERSP>
</MESSAGE>
</CIM>

@KSchopmeyer
Copy link
Collaborator Author

This is going to take more than a day to fix.

KSchopmeyer added a commit that referenced this issue Nov 10, 2019
The first error encountered was that in the rollback logic the instance
paths are not set into instances when they are created locally. This is
because the normal compile with the pyebem mof compiler does not force
this.  We cannot do this by forcing the Alias on every instance created
in the mof to be able to remove it so the solution is to create a new
MOFWBEMConnection adaption that set the path into the mof as it is
created as part of the remove.

Added a test to test_mof_compiler.py to assure that we are propagating
key property qualifiers correctly because that appears to be another
issue in the remove.
@andy-maier
Copy link
Contributor

Will be fixed with PR #1944 (for master) and TBD for 0.15.0.

@andy-maier andy-maier removed discussion needed roll back needed Roll back into latest fix branch is needed labels Nov 11, 2019
KSchopmeyer added a commit that referenced this issue Nov 15, 2019
The first error encountered was that in the rollback logic the instance
paths are not set into instances when they are created locally. This is
because the normal compile with the pyebem mof compiler does not force
this.  We cannot do this by forcing the Alias on every instance created
in the mof to be able to remove it so the solution is to create a new
MOFWBEMConnection adaption that set the path into the mof as it is
created as part of the remove.

Added a test to test_mof_compiler.py to assure that we are propagating
key property qualifiers correctly because that appears to be another
issue in the remove.
@KSchopmeyer
Copy link
Collaborator Author

KSchopmeyer commented Nov 15, 2019

DISCUSSION: Found two problems (see above) Fixed first but have not figured out why the second is happening. The question is, whether this is worthwhile for 0.15.0 since it has been broken for many years and nobody noticed.

I propose that If I cannot define a fix for the second problem (association being created as part of the remove process with incomplete reference property (keybindings empty) when all other 0,15.5 release is ready, we commit what is fixed and redo bug.

Note that there are no tests for this, or even for the mof_compiler script.

KSchopmeyer added a commit that referenced this issue Nov 22, 2019
The first error encountered was that in the rollback logic the instance
paths are not set into instances when they are created locally. This is
because the normal compile with the pyebem mof compiler does not force
this.  We cannot do this by forcing the Alias on every instance created
in the mof to be able to remove it so the solution is to create a new
MOFWBEMConnection adaption that set the path into the mof as it is
created as part of the remove.

Added a test to test_mof_compiler.py to assure that we are propagating
key property qualifiers correctly because that appears to be another
issue in the remove.
KSchopmeyer added a commit that referenced this issue Nov 22, 2019
The first error encountered was that in the rollback logic the instance
paths are not set into instances when they are created locally. This is
because the normal compile with the pyebem mof compiler does not force
this.  We cannot do this by forcing the Alias on every instance created
in the mof to be able to remove it so the solution is to create a new
MOFWBEMConnection adaption that set the path into the mof as it is
created as part of the remove.

The second issue found was that the property values for reference
properties in associations where the value is defined by the mof
compiler alias were being inserted into the instance with keybindings
with no keys.

Corrects this issue in mof_compiler.py; incorrect parameter on GetClass
was being passed to recursive GetClass call that could result in
compiler processing classes for key qualifier but the getclass with
IncludeQualifiers=False meant no keys were found. Thus, reference
properties in instances where value defined by alais were
built with empty keybindings.

Added a test to test_mof_compiler.py to assure that we are propagating
key property qualifiers correctly.

Adds a bash script to test mof_compiler

As a temporary measure, this adds a script that executes mof_compiler
multiple time with the same MOF to add the MOF to a defined server,
remove it and do that a second time.

This is a very limited test using a local WBEM Server and is in the
manualtests group.
@andy-maier
Copy link
Contributor

Fixed in PR #1944 for master, and TBD for 0.15.0.

@andy-maier andy-maier added resolution: fixed roll back needed Roll back into latest fix branch is needed and removed discussion needed under work labels Nov 24, 2019
KSchopmeyer added a commit that referenced this issue Nov 26, 2019
The first error encountered was that in the rollback logic the instance
paths are not set into instances when they are created locally. This is
because the normal compile with the pyebem mof compiler does not force
this.  We cannot do this by forcing the Alias on every instance created
in the mof to be able to remove it so the solution is to create a new
MOFWBEMConnection adaption that set the path into the mof as it is
created as part of the remove.

The second issue found was that the property values for reference
properties in associations where the value is defined by the mof
compiler alias were being inserted into the instance with keybindings
with no keys.

Corrects this issue in mof_compiler.py; incorrect parameter on GetClass
was being passed to recursive GetClass call that could result in
compiler processing classes for key qualifier but the getclass with
IncludeQualifiers=False meant no keys were found. Thus, reference
properties in instances where value defined by alais were
built with empty keybindings.

Added a test to test_mof_compiler.py to assure that we are propagating
key property qualifiers correctly.

Adds a bash script to test mof_compiler

As a temporary measure, this adds a script that executes mof_compiler
multiple time with the same MOF to add the MOF to a defined server,
remove it and do that a second time.

This is a very limited test using a local WBEM Server and is in the
manualtests group.
KSchopmeyer added a commit that referenced this issue Nov 29, 2019
Fixes issue in mof_compiler where the remove option fails to remove
mof created by an earlier compile of the same file.

Adds a test for this to test_mofcompiler.py based on using the
mocker and test.mof
KSchopmeyer added a commit that referenced this issue Nov 29, 2019
Fixes issue in mof_compiler where the remove option fails to remove
mof created by an earlier compile of the same file. This was
primarily an issue of getting the instance paths correct on the
CreateInstance.

Adds a test for rollbacik to test_mofcompiler.py based on using the
mocker and test.mof

Adds a second manual test against a server at http://localhost
that executes mof_compiler to create and rollback multiple times
in two namespaces.

Adds documentation on the rollback limitations to the rollback
method documentation.
andy-maier pushed a commit that referenced this issue Nov 29, 2019
Fixes issue in mof_compiler where the remove option fails to remove
mof created by an earlier compile of the same file. This was
primarily an issue of getting the instance paths correct on the
CreateInstance.

Adds a test for rollbacik to test_mofcompiler.py based on using the
mocker and test.mof

Adds a second manual test against a server at http://localhost
that executes mof_compiler to create and rollback multiple times
in two namespaces.

Adds documentation on the rollback limitations to the rollback
method documentation.
@andy-maier andy-maier added roll back/forward done Roll back or roll forward has been done and removed roll back needed Roll back into latest fix branch is needed labels Nov 29, 2019
andy-maier pushed a commit that referenced this issue Nov 29, 2019
Fixes issue in mof_compiler where the remove option fails to remove
mof created by an earlier compile of the same file. This was
primarily an issue of getting the instance paths correct on the
CreateInstance.

Adds a test for rollbacik to test_mofcompiler.py based on using the
mocker and test.mof

Adds a second manual test against a server at http://localhost
that executes mof_compiler to create and rollback multiple times
in two namespaces.

Adds documentation on the rollback limitations to the rollback
method documentation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants