LDAP state module improvements #62791

rhansen · 2022-10-01T03:37:54Z

What does this PR do?

Various improvements to the ldap state module and ldap3 execution module:

Redo and expand tests.
Documentation improvements.
Code health improvements.
Improvements to attribute value ordering in ldap.managed.
Improvements to encoding/decoding of attribute values.
Work around a flaw in the way python-ldap generates modification lists.

There are now integration tests, which requires OpenLDAP to be installed. The tests use a custom Docker image to get OpenLDAP (see saltstack/salt-ci-containers#3).

What issues does this PR fix or reference?

None.

Merge requirements satisfied?

[NOTICE] Bug fixes or features added to Salt require tests.

Docs
Changelog - https://docs.saltproject.io/en/master/topics/development/changelog.html
Tests written/updated

Commits signed with GPG?

No

Please review Salt's Contributing Guide for best practices.

See GitHub's page on GPG signing for more information about signing commits with GPG.

MKLeb · 2022-10-01T05:56:37Z

Hi @rhansen, thanks for your contribution!

It looks like there are some failing tests here. And for the integration tests, perhaps use a docker container for openldap? Something like https://hub.docker.com/r/bitnami/openldap/ should work, and there are some examples around the test suite where we use pytest-salt-factories to spin up docker containers in our test suite!

MKLeb · 2022-10-01T05:58:44Z

Here is an example of using saltfactories for the docker containers -

salt/tests/pytests/functional/cache/test_consul.py

Line 41 in 18e878d

def consul_container(salt_factories):

rhansen · 2022-10-02T04:53:06Z

Thanks for the pointer @MKLeb.

I'm having a bit of trouble figuring out how to set up tests. The python-ldap package must be installed on a machine with OpenLDAP installed because it compiles some C code against an OpenLDAP library. I think the most straightforward way to handle this is to create a custom Docker container that is both a minion and an OpenLDAP server, and control the minion from a test master daemon. Any tips on how to do that? (e.g., pointers to existing test code that does something similar) Or do you have an alternative suggestion?

MKLeb · 2022-10-03T16:07:22Z

Thanks for the pointer @MKLeb.

I'm having a bit of trouble figuring out how to set up tests. The python-ldap package must be installed on a machine with OpenLDAP installed because it compiles some C code against an OpenLDAP library. I think the most straightforward way to handle this is to create a custom Docker container that is both a minion and an OpenLDAP server, and control the minion from a test master daemon. Any tips on how to do that? (e.g., pointers to existing test code that does something similar) Or do you have an alternative suggestion?

Apologies, I should have looked a bit further here. It seems OpenLDAP is already installed into the CI machines (see here).

From what I can tell, your integration tests are being skipped because it can't find the ldapadd command. Perhaps we can use the ldap3 module itself (and its add method) to add the entries instead of ldapadd from slapdtest? It all depends what underlying tools it uses to do that. I get that it's kind of meta to use the ldap3 mod to test it, but I think that's our best and quickest route here as long as it doesn't rely on ldapadd under the hood.

rhansen · 2022-10-04T21:46:05Z

It seems OpenLDAP is already installed into the CI machines (see here).

It looks like libldap2-dev (on Debian-based systems anyway) needs to be installed as well:

  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DHAVE_SASL -DHAVE_TLS -DLDAPMODULE_VERSION=3.4.3 "-DLDAPMODULE_AUTHOR=python-ldap project" "-DLDAPMODULE_LICENSE=Python style" -IModules -I/__w/salt/salt/.nox/docs-html-compress-false-clean-true/include -I/usr/local/include/python3.8 -c Modules/LDAPObject.c -o build/temp.linux-x86_64-cpython-38/Modules/LDAPObject.o
  In file included from Modules/LDAPObject.c:3:
  Modules/common.h:15:10: fatal error: lber.h: No such file or directory
   #include <lber.h>
            ^~~~~~~~
  compilation terminated.
  error: command '/usr/bin/gcc' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for python-ldap

It's probably easy to add that package to the golden images, but I think it would be nicer if the tests tested against a Docker container rather than requiring OpenLDAP to be installed on the CI machines. Getting that to work will take some time and effort; I'll start working on that. If you'd prefer, we can do the easy thing first (install libldap2-dev on the CI machines) and migrate the tests to Docker in a future pull request.

MKLeb · 2022-10-05T00:15:41Z

I agree, docker would be the nice way to do it, and likely the correct way as well. There is an example of how we run a minion in a docker container here. It uses an image built in https://github.com/saltstack/salt-ci-containers. We can't do quite what those do in that repo though, because we likely want to test against a minion running the changes on the current branch, as is the case here for instance. You can probably just pip install the local salt, all the ldap dependencies, and pip install python-ldap. I would imagine you would either create an inline dockerfile and build it each time or create a minimal dockerfile that runs the minion on the local salt and then use that minion to then install the ldap dependencies and python-ldap. Not sure which is better honestly, but for no reason at all I'm leaning towards the second option :)

Here is the docs on how the container minion works. Let me know if you need any help!

rhansen · 2022-10-06T17:39:20Z

OK, I think I got it working. It depends on saltstack/salt-ci-containers#3 so the tests will fail until that is merged and an image is published.

rhansen · 2022-10-06T20:34:20Z

I'll rebase onto latest master and fix some of the lint errors.

rhansen · 2022-10-06T22:56:16Z

Looks like the new openldap-minion-* container is leaking into the tests.pytests.integration.netapi.* tests. The openldap_minion fixture has scope package, so it should be shut down by the time those tests run. Any ideas?

MKLeb · 2022-10-07T03:16:59Z

Looks like the new openldap-minion-* container is leaking into the tests.pytests.integration.netapi.* tests. The openldap_minion fixture has scope package, so it should be shut down by the time those tests run. Any ideas?

Hey @rhansen, thanks for diving deep into this! From a cursory glance, I'm not sure what is going on there. Maybe I can dig around here tomorrow and look for a fix. I'll let you know as soon as I do.

MKLeb · 2022-10-07T19:19:54Z

Aha! I got it to work. I switched them to module scopes, and also added a line to delete the minion's key after we are done with it (that is what was messing up the netapi tests). Here is my diff, and you'll notice I switched from using salt_cilent to salt_cli just to be a little more explicit and consisten across the test suite.

diff --git a/tests/pytests/integration/modules/test_ldap3.py b/tests/pytests/integration/modules/test_ldap3.py
index e45d6617be..b591636dc5 100644
--- a/tests/pytests/integration/modules/test_ldap3.py
+++ b/tests/pytests/integration/modules/test_ldap3.py
@@ -11,7 +11,7 @@ pytestmark = [
 
 
 def test_search(openldap_minion_run, subtree, u0dn):
-    assert openldap_minion_run("ldap3.search", base=subtree) == {
+    assert openldap_minion_run("ldap3.search", base=subtree).data == {
         subtree: {
             "objectClass": ["dcObject", "organization"],
             "dc": ["test_search"],
@@ -29,7 +29,7 @@ def test_search(openldap_minion_run, subtree, u0dn):
 def test_search_filter(openldap_minion_run, subtree, u0dn):
     assert openldap_minion_run(
         "ldap3.search", base=subtree, filterstr="(sn=Lastname)"
-    ) == {
+    ).data == {
         u0dn: {
             "objectClass": ["person"],
             "cn": ["u0"],
diff --git a/tests/support/pytest/ldap.py b/tests/support/pytest/ldap.py
index 6ec208bb79..2d470b1c3c 100644
--- a/tests/support/pytest/ldap.py
+++ b/tests/support/pytest/ldap.py
@@ -202,8 +202,8 @@ class SlapdMinion(SaltMinion):
         self.__run_unpriv(cmd)
 
 
-@pytest.fixture(scope="package")
-def openldap_minion(salt_master, salt_client):
+@pytest.fixture(scope="module")
+def openldap_minion(salt_master, salt_cli, salt_key_cli):
     name = random_string("openldap-minion-")
     c = salt_master.salt_minion_daemon(
         name,
@@ -218,15 +218,15 @@ def openldap_minion(salt_master, salt_client):
     # keyword argument 'python_executable'".
     c.python_executable = "python3"
     with c.started():
-        ret = salt_client.run("test.ping", minion_tgt=c.id)
-        assert ret is True
+        ret = salt_cli.run("test.ping", minion_tgt=c.id)
+        assert ret.data is True
         yield c
+    salt_key_cli.run("-y", "-d", c.id)
 
-
-@pytest.fixture(scope="package")
-def openldap_minion_run(openldap_minion, salt_client):
+@pytest.fixture(scope="module")
+def openldap_minion_run(openldap_minion, salt_cli):
     def _run(fn, *args, **kwargs):
-        return salt_client.run(
+        return salt_cli.run(
             fn,
             *args,
             minion_tgt=openldap_minion.id,
@@ -237,8 +237,8 @@ def openldap_minion_run(openldap_minion, salt_client):
     yield _run
 
 
-@pytest.fixture(scope="package")
-def openldap_minion_apply(openldap_minion_run, salt_client):
+@pytest.fixture(scope="module")
+def openldap_minion_apply(openldap_minion_run):
     def _apply(fn, **kwargs):
         has_name = "name" in kwargs
         kwargs.setdefault("name", "x")

rhansen · 2022-10-07T23:41:05Z

Aha! I got it to work.

Awesome, thank you! I rebased and pushed the fixes; hopefully the tests will all pass.

I switched them to module scopes,

Oops, I conflated "package" with "module" again. I meant to do module, but thought it was spelled "package". 🤦

and also added a line to delete the minion's key after we are done with it (that is what was messing up the netapi tests).

Ah, makes sense! I had expected the master to be cleaned up after module scope (due to confusing package with module) so I didn't think I needed to clean up the key.

I switched from using salt_cilent to salt_cli just to be a little more explicit and consistent across the test suite.

I intentionally used salt_client because tuples can't be serialized to JSON, and I wanted to pass a non-list object to make sure that it can accept arbitrary iterables. I added a comment to make that clear, though perhaps it would be better to do the non-list iterable check in a unit test instead?

Thanks for the help!

MKLeb · 2022-10-10T17:24:57Z

I intentionally used salt_client because tuples can't be serialized to JSON, and I wanted to pass a non-list object to make sure that it can accept arbitrary iterables. I added a comment to make that clear, though perhaps it would be better to do the non-list iterable check in a unit test instead?

Oops, I missed that comment. If you wanted to make that a unit test that should be fine! Also, is there a reason you deleted the old ldap module unit test file? If they were incompatible with your changes, could you add a few more simple ones to make sure we have some coverage there as well? Thanks! Let me know if you need any more help.

Future commits will modify this class to fix some bugs.

Move all encoding and decoding logic to the `AttributeValueSet` class and consistently use it when reading from or writing to LDAP. This fixes some encoding/decoding corner cases, and improves the API's usability. Technically this is a backwards-incompatible change: `ldap3.search` and `ldap.managed` now return decoded strings when possible. However, it appears that the Salt master (or maybe the master/minion protocol?) automatically decodes returned `bytes` objects to `str` when possible, so this change only affects direct function calls.

rhansen · 2022-10-26T05:36:33Z

Figured it out: saltstack/pytest-salt-factories#139

I still want to add some more unit tests, and #62932 needs to be merged. Once those are done, this should be ready to go.

MKLeb · 2022-10-26T16:43:45Z

Sorry for not getting back to you quickly, but glad to hear you figured it out.

Figured it out: saltstack/pytest-salt-factories#139

I still want to add some more unit tests, and #62932 needs to be merged. Once those are done, this should be ready to go.

Going to look at #62932 today and suggest changes (if any). We don't need to solve that salt factories issue first, correct?

rhansen · 2022-10-26T18:54:33Z

We don't need to solve that salt factories issue first, correct?

Correct. I have a workaround in place.

s0undt3ch · 2022-11-04T06:26:48Z

tests/support/pytest/ldap.py

+                # Bind mount the checked-out source code into the Docker
+                # container so that a minion daemon started inside the container
+                # will run the code to be tested.
+                RUNTIME_VARS.CODE_DIR: {


Don't use RUNTIME_VARS, that's something we inherited from the old test suite.

from tests.conftest import CODE_DIR

s0undt3ch · 2022-11-04T06:30:54Z

tests/support/pytest/ldap.py

+        v = importlib_metadata.version("pytest-salt-factories")
+        for cmd in [
+            ("install_packages", "python3-pip"),
+            ("pip", "install", f"pytest-salt-factories=={v}"),


Salt factories is bind mounted on the minion container

https://github.com/saltstack/pytest-salt-factories/blob/1.0.0rc20/src/saltfactories/daemons/container.py#L772

It's not importable but log forwarding "should" still work.

s0undt3ch · 2022-11-04T06:34:41Z

tests/pytests/integration/modules/test_ldap3.py

+pytest_plugins = [
+    "tests.support.pytest.ldap",
+]


Suggested change

pytest_plugins = [

"tests.support.pytest.ldap",

]

from tests.support.pytest.ldap import openldap_minion_run

Import the fixture instead.

dwoz · 2023-12-11T20:50:01Z

Closing this due to inactivity. Anyone should feel free to re-open it if they want to see it through to the end in one release cycle.

rhansen requested a review from a team as a code owner October 1, 2022 03:37

rhansen requested review from MKLeb and removed request for a team October 1, 2022 03:37

rhansen changed the title ~~Fix openldap x ordered~~ LDAP improvements Oct 1, 2022

rhansen force-pushed the fix-openldap-x-ordered branch 2 times, most recently from a92293d to af7e7c7 Compare October 4, 2022 21:27

rhansen mentioned this pull request Oct 6, 2022

Add custom openldap-minion image saltstack/salt-ci-containers#3

Merged

rhansen force-pushed the fix-openldap-x-ordered branch from af7e7c7 to b1dad6e Compare October 6, 2022 17:38

rhansen force-pushed the fix-openldap-x-ordered branch from 3d5372b to 5f93900 Compare October 6, 2022 21:17

rhansen force-pushed the fix-openldap-x-ordered branch from 5f93900 to 772e3e7 Compare October 6, 2022 22:57

rhansen force-pushed the fix-openldap-x-ordered branch from 772e3e7 to d48cee0 Compare October 7, 2022 03:49

rhansen force-pushed the fix-openldap-x-ordered branch from d48cee0 to 4cdf713 Compare October 7, 2022 23:38

rhansen force-pushed the fix-openldap-x-ordered branch from 4cdf713 to 55d7132 Compare October 8, 2022 01:13

rhansen changed the title ~~LDAP improvements~~ LDAP state module improvements Oct 8, 2022

rhansen force-pushed the fix-openldap-x-ordered branch 3 times, most recently from 20d9858 to 3c09fc9 Compare October 8, 2022 22:08

rhansen added 5 commits October 26, 2022 01:32

ldap: Define a new class for holding attribute values

e33d167

Future commits will modify this class to fix some bugs.

ldap: Preserve the order of multi-valued attributes

2c45dec

ldap: Ignore order when checking equality of attribute value sets

2f1ee6b

ldap: Redo change() to avoid flaw in python-ldap package

89b405e

rhansen force-pushed the fix-openldap-x-ordered branch from 3a94258 to 89b405e Compare October 26, 2022 05:34

s0undt3ch requested changes Nov 4, 2022

View reviewed changes

dwoz closed this Dec 11, 2023

dwoz added help-wanted Community help is needed to resolve this Abandoned labels Dec 11, 2023

dwoz temporarily deployed to ci December 11, 2023 21:07 — with GitHub Actions Inactive

dwoz temporarily deployed to ci December 11, 2023 21:08 — with GitHub Actions Inactive

dwoz temporarily deployed to ci December 12, 2023 06:16 — with GitHub Actions Inactive

dwoz temporarily deployed to ci December 12, 2023 06:17 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LDAP state module improvements #62791

LDAP state module improvements #62791

rhansen commented Oct 1, 2022 •

edited

MKLeb commented Oct 1, 2022

MKLeb commented Oct 1, 2022

rhansen commented Oct 2, 2022

MKLeb commented Oct 3, 2022 •

edited

rhansen commented Oct 4, 2022

MKLeb commented Oct 5, 2022

rhansen commented Oct 6, 2022

rhansen commented Oct 6, 2022

rhansen commented Oct 6, 2022

MKLeb commented Oct 7, 2022

MKLeb commented Oct 7, 2022

rhansen commented Oct 7, 2022 •

edited

MKLeb commented Oct 10, 2022

rhansen commented Oct 26, 2022

MKLeb commented Oct 26, 2022

rhansen commented Oct 26, 2022

s0undt3ch Nov 4, 2022

s0undt3ch Nov 4, 2022

s0undt3ch Nov 4, 2022

dwoz commented Dec 11, 2023

LDAP state module improvements #62791

LDAP state module improvements #62791

Conversation

rhansen commented Oct 1, 2022 • edited

What does this PR do?

What issues does this PR fix or reference?

Merge requirements satisfied?

Commits signed with GPG?

MKLeb commented Oct 1, 2022

MKLeb commented Oct 1, 2022

rhansen commented Oct 2, 2022

MKLeb commented Oct 3, 2022 • edited

rhansen commented Oct 4, 2022

MKLeb commented Oct 5, 2022

rhansen commented Oct 6, 2022

rhansen commented Oct 6, 2022

rhansen commented Oct 6, 2022

MKLeb commented Oct 7, 2022

MKLeb commented Oct 7, 2022

rhansen commented Oct 7, 2022 • edited

MKLeb commented Oct 10, 2022

rhansen commented Oct 26, 2022

MKLeb commented Oct 26, 2022

rhansen commented Oct 26, 2022

s0undt3ch Nov 4, 2022

Choose a reason for hiding this comment

s0undt3ch Nov 4, 2022

Choose a reason for hiding this comment

s0undt3ch Nov 4, 2022

Choose a reason for hiding this comment

dwoz commented Dec 11, 2023

rhansen commented Oct 1, 2022 •

edited

MKLeb commented Oct 3, 2022 •

edited

rhansen commented Oct 7, 2022 •

edited