Issue/6025 implement discovery handler #6264

Hugo-Inmanta · 2023-07-12T14:03:22Z

Description

implement DiscoveryHandler
Verify that the Id constructor and resource_str method are part of the stable API and documented.

Part of #6025

Self Check:

Strike through any lines that are not applicable (~~line~~) then check the box

Attached issue to pull request
Changelog entry
Type annotations are present
Code is clear and sufficiently documented
No (preventable) type errors (check using make mypy or make mypy-diff)
Sufficient test cases (reproduces the bug/tests the requested feature)
Correct, in line with design
End user documentation is included or an issue is created for end-user documentation (add ref to issue here: )
If this PR fixes a race condition in the test suite, also push the fix to the relevant stable branche(s) (see test-fixes for more info)

…6025-implement-discovery-handler

Hugo-Inmanta · 2023-07-13T13:55:45Z

src/inmanta/agent/handler.py

+    # The DiscoveryHandler is generic in both the handler's Discovery Resource type (DRT)
+    # and the Unmanaged Resource type (URT) it reports to the server. The second has to be serializable.
+
+    # This class deploys instances of DRT and reports URT to the server.


deploys instances of DRT
I'm not sure this is effectively the case with my implementation

Indeed, this is still missing. The main resource entry-point (at the moment, depends very slightly on the final outcome of the other PR) is deploy so that method should be implemented in this class to call discover_resources and report_discovered_resources.

Hugo-Inmanta · 2023-07-13T14:01:00Z

src/inmanta/agent/handler.py

+    def discover_resources(self, ctx: HandlerContext, discovery_resource: DRT) -> abc.Mapping[ResourceIdStr, URT]:
+        raise NotImplementedError
+
+    def report_discovered_resources(self, ctx: HandlerContext, resource: DRT) -> None:


I'm unsure of whether this logic should happen in a synchronous or asynchronous context.
The testcase currently uses the async version of this (async_report_discovered_resources) and passes.
I was having second thoughts about it and tried the synchronous version (report_discovered_resources) and it looks like it is hanging somewhere.

All agent ABC methods are synchronous so this should be synchronous as well. I would even go as far as to drop the async method. For testing purposes, using the sync client is more complicated because if both server and client run on the same thread, a synchronous call would hang the server so it can never respond. I'm pretty sure we have some fixtures to work with that, I'll add a suggestion to your test cases.

Hugo-Inmanta · 2023-07-13T14:11:34Z

tests/agent_server/test_discovery_handler.py

+    # resource_handler.report_discovered_resources(ctx, discovery_resource)
+    await resource_handler.async_report_discovered_resources(ctx, discovery_resource)


See comment on the method themselves, I'm not confident about that part

sanderr

Intermediate review: I haven't looked at test cases or inmanta.resources yet.

src/inmanta/agent/handler.py

sanderr · 2023-07-18T09:07:18Z

src/inmanta/agent/handler.py

+    def discover_resources(self, ctx: HandlerContext, discovery_resource: DRT) -> abc.Mapping[ResourceIdStr, URT]:
+        raise NotImplementedError
+
+    def report_discovered_resources(self, ctx: HandlerContext, resource: DRT) -> None:


This should be deploy. Could optionally be split into discover -> serialize -> report, so deploy just has to chain them together. No strong preference on the last part, unless the serialization turns out to be very verbose.

sanderr · 2023-07-18T09:08:25Z

src/inmanta/agent/handler.py

+        try:
+            self.pre(ctx, resource)
+            # report to the server
+            discovered_resources: List[DiscoveredResource] = [


Should be typed as lowercase list or abc.Sequence. You're not modifying it so abc.Sequence presents the clearer picture.

sanderr · 2023-07-18T09:10:46Z

src/inmanta/agent/handler.py

+            self.pre(ctx, resource)
+            # report to the server
+            discovered_resources: List[DiscoveredResource] = [
+                DiscoveredResource(discovered_resource_id=resource_id, values=values)


These types do not match, we should serialize here.

sanderr · 2023-07-18T09:20:51Z

src/inmanta/agent/handler.py

+                    ),
+                    urt=str(URT),
+                    resource_id=resource.id,
+                    exception=f"{e.__class__.__name__}('{e}')",


I think repr() does pretty much this:

>>> e = Exception("hellow orld") >>> str(e) 'hellow orld' >>> repr(e) "Exception('hellow orld')"

Same for other message

Ah turns out it is not the same regarding formatting:

e = Exception("An\nError\tMessage") print(str(e)) print(repr(e)) print(f"{e.__class__.__name__}('{e}')")

An Error Message Exception('An\nError\tMessage') Exception('An Error Message')

And this test fails if we change it to repr

Interesting. To be honest I personally prefer the repr output but if we were already using the other approach (I hadn't realized when I commented) I guess we should stick with that.

src/inmanta/agent/handler.py

Co-authored-by: Sander Van Balen <git@sandervanbalen.be>

…6025-implement-discovery-handler

…vel-handler-ABC

…6025-implement-discovery-handler

Co-authored-by: arnaudsjs <2684622+arnaudsjs@users.noreply.github.com>

…6025-implement-discovery-handler

Hugo-Inmanta · 2023-08-02T13:30:33Z

Still a couple of mypy errors that I didn't manage to handle on my own:

+ src/inmanta/agent/handler.py:1018: error: Argument 2 of "execute" is incompatible with supertype "HandlerAPI"; supertype defines the argument type as "Resource"  [override]
+ src/inmanta/agent/handler.py:1036: error: Argument 2 of "deploy" is incompatible with supertype "HandlerAPI"; supertype defines the argument type as "Resource"  [override]
+ src/inmanta/agent/handler.py:1036: note: This violates the Liskov substitution principle
+ src/inmanta/agent/handler.py:1036: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
+ src/inmanta/agent/handler.py:1071: error: Argument 2 of "execute" is incompatible with supertype "HandlerAPI"; supertype defines the argument type as "Resource"  [override]
+ src/inmanta/agent/handler.py:1071: note: This violates the Liskov substitution principle
+ src/inmanta/agent/handler.py:1071: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
+ src/inmanta/agent/handler.py:1089: error: Argument 1 to "json_encode" has incompatible type "D"; expected "Union[Union[BaseModel, Union[UUID, StrictNonIntBool, int, float, datetime, str]], Sequence[Union[BaseModel, Union[UUID, StrictNonIntBool, int, float, datetime, str]]], Mapping[str, Union[BaseModel, Union[UUID, StrictNonIntBool, int, float, datetime, str]]], None]"  [arg-type]

wouterdb

Looks good overall

wouterdb · 2023-08-02T13:47:13Z

src/inmanta/agent/handler.py

+        resource: R,
+        requires: abc.Mapping[ResourceIdStr, ResourceState],
+    ) -> None:
+        self.execute(ctx, resource)


We may want to copy over some of the this code

resources_in_unexpected_state = filter_resources_in_unexpected_state(requires) if resources_in_unexpected_state: ctx.set_status(const.ResourceState.skipped) ctx.warning( "Resource %(resource)s skipped because a dependency is in an unexpected state: %(unexpected_states)s", resource=resource.id.resource_version_str(), unexpected_states=str({rid: state.value for rid, state in resources_in_unexpected_state.items()}), ) return failed_dependencies = [req for req, status in requires.items() if status != ResourceState.deployed] if not any(failed_dependencies): self.execute(ctx, resource) if _should_reload(): self.do_reload(ctx, resource) else: ctx.set_status(const.ResourceState.skipped) ctx.info( "Resource %(resource)s skipped due to failed dependencies: %(failed)s", resource=resource.id.resource_version_str(), failed=str(failed_dependencies), )

To handle skip and failed dependencies

I lifted this logic in the HandlerAPI's deploy in the other PR

wouterdb · 2023-08-02T13:48:51Z

src/inmanta/agent/handler.py

+            # serialize resources and report to the server
+            discovered_resources: abc.Sequence[DiscoveredResource] = [
+                DiscoveredResource(discovered_resource_id=resource_id, values=json.loads(json_encode(values)))
+                for resource_id, values in self.discover_resources(ctx, resource).items()


purely a matter of taste, but I would put the self.discover_resources(ctx, resource) on a line of its own, as it is the main thing here. It is kind of hidden now.

wouterdb · 2023-08-02T13:49:41Z

src/inmanta/agent/handler.py

+        try:
+            self.pre(ctx, resource)
+            # serialize resources and report to the server
+            discovered_resources: abc.Sequence[DiscoveredResource] = [


If we run a dryrun, do we want to execute all this?

wouterdb · 2023-08-02T13:52:28Z

tests/agent_server/test_discovery_handler.py

+        def discover_resources(
+            self, ctx: HandlerContext, discovery_resource: MyDiscoveryResource
+        ) -> abc.Mapping[ResourceIdStr, MyUnmanagedResource]:
+            dirs = os.listdir(self._top_dir_path)


Why files, why not just an array of strings?

wouterdb · 2023-08-02T13:53:29Z

tests/agent_server/test_discovery_handler.py

+from inmanta.resources import DiscoveryResource, Id, resource
+
+
+async def test_discovery_resource_handler(


I would also add tests for dryrun, get_fact and a few requires-provides scenarios.

…#6025, PR #6415) # Description - [x] implement `DiscoveryHandler` - [x] Verify that the `Id` constructor and `resource_str` method are part of the stable API and documented. This PR replaces #6264 Part of #6025 # Self Check: - [x] Attached issue to pull request - [ ] Changelog entry - [x] Type annotations are present - [x] Code is clear and sufficiently documented - [x] No (preventable) type errors (check using make mypy or make mypy-diff) - [x] Sufficient test cases (reproduces the bug/tests the requested feature) - [x] Correct, in line with design - [x] End user documentation is included or an issue is created for end-user documentation (#6270) - [ ] ~~If this PR fixes a race condition in the test suite, also push the fix to the relevant stable branche(s) (see [test-fixes](https://internal.inmanta.com/development/core/tasks/build-master.html#test-fixes) for more info)~~

Hugo-Inmanta added 19 commits July 7, 2023 08:35

[WIP]

b55d152

[FIX] fix broken anchor in link

cad6ced

[WIP]

31dec93

[WIP] Implement top-level handler ABC

8dba6f1

[FIX] remove unrelated change

769ab33

[WIP]

5a6919f

[WIP]

15856ba

[FIX]

61a75cb

Merge branch 'issue/6025-implement-top-level-handler-ABC' into issue/…

1b0b7c9

…6025-implement-discovery-handler

[DOC] add documentation for Id and resource_str

cfc0ce3

[WIP]

9e010ca

[WIP]

247b5e9

[WIP]

7f02a3c

[WIP]

1357fef

Merge branch 'master' into issue/6025-implement-discovery-handler

69f5439

[WIP]

6996342

[WIP]

9b75118

[WIP]

8f0092d

[GIT] move documentation to a separate PR

1f010bb

Hugo-Inmanta requested a review from sanderr July 13, 2023 14:13

Hugo-Inmanta commented Jul 17, 2023

View reviewed changes

Hugo-Inmanta added 2 commits July 17, 2023 11:17

[FORMAT]

af7fbdd

[FIX] PEP8

4549578

sanderr requested changes Jul 18, 2023

View reviewed changes

Hugo-Inmanta and others added 6 commits July 19, 2023 09:34

Update src/inmanta/agent/handler.py

5bcc5e4

Co-authored-by: Sander Van Balen <git@sandervanbalen.be>

[FIX] fix typo

9819c83

[WIP]

466dfda

[WIP]

e6c65cd

Merge branch 'master' into issue/6025-implement-top-level-handler-ABC

982db23

[FORMAT]

d1b54b0

Hugo-Inmanta added 9 commits July 24, 2023 09:02

[DOC]

806acd1

[WIP]

01023e6

Merge branch 'issue/6025-implement-top-level-handler-ABC' into issue/…

5c62a8a

…6025-implement-discovery-handler

Merge branch 'master' into issue/6025-implement-discovery-handler

e863677

[WIP]

cf516ae

[WIP]

ed99bd0

[WIP]

3357e12

[WIP]

8e894c2

[WIP]

8127220

Hugo-Inmanta requested review from sanderr and wouterdb July 25, 2023 12:23

wouterdb and others added 13 commits July 31, 2023 13:11

proposal for handler api class

806e730

Merge branch 'issue/handler_abc_wdb' into issue/6025-implement-top-le…

2f3555b

…vel-handler-ABC

[WIP]

f3473eb

Merge branch 'issue/6025-implement-top-level-handler-ABC' into issue/…

d99ceee

…6025-implement-discovery-handler

[WIP]

72a61bb

Update src/inmanta/agent/handler.py

4c97da5

Co-authored-by: arnaudsjs <2684622+arnaudsjs@users.noreply.github.com>

Merge branch 'issue/6025-implement-top-level-handler-ABC' into issue/…

90f7509

…6025-implement-discovery-handler

[WIP]

221a0f0

[WIP]

f1f37ed

Merge branch 'master' into issue/6025-implement-discovery-handler

1fa818d

[WIP]

be04e9a

Merge branch 'master' into issue/6025-implement-discovery-handler

334fb7f

[FIX] fix bound type

ebb3580

wouterdb requested changes Aug 2, 2023

View reviewed changes

arnaudsjs mentioned this pull request Aug 21, 2023

Issue/6025 implement discovery handler (2) #6415

Closed

11 tasks

arnaudsjs closed this Aug 21, 2023

arnaudsjs deleted the issue/6025-implement-discovery-handler branch August 21, 2023 06:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue/6025 implement discovery handler #6264

Issue/6025 implement discovery handler #6264

Hugo-Inmanta commented Jul 12, 2023 •

edited

Hugo-Inmanta Jul 13, 2023

sanderr Jul 18, 2023

Hugo-Inmanta Jul 13, 2023

sanderr Jul 18, 2023

Hugo-Inmanta Jul 13, 2023

sanderr left a comment

sanderr Jul 18, 2023

sanderr Jul 18, 2023

sanderr Jul 18, 2023

sanderr Jul 18, 2023

Hugo-Inmanta Jul 21, 2023

sanderr Jul 28, 2023

Hugo-Inmanta commented Aug 2, 2023

wouterdb left a comment

wouterdb Aug 2, 2023

Hugo-Inmanta Aug 3, 2023

wouterdb Aug 2, 2023

wouterdb Aug 2, 2023

wouterdb Aug 2, 2023

wouterdb Aug 2, 2023

		# resource_handler.report_discovered_resources(ctx, discovery_resource)
		await resource_handler.async_report_discovered_resources(ctx, discovery_resource)

		from inmanta.resources import DiscoveryResource, Id, resource


		async def test_discovery_resource_handler(

Issue/6025 implement discovery handler #6264

Issue/6025 implement discovery handler #6264

Conversation

Hugo-Inmanta commented Jul 12, 2023 • edited

Description

Self Check:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanderr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hugo-Inmanta commented Aug 2, 2023

wouterdb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hugo-Inmanta commented Jul 12, 2023 •

edited