[Extensibility] Zeroconf #163

DazWilkin · 2020-12-08T22:13:16Z

Implements: #132

What this PR does / why we need it:

Provides a Linux-only Zeroconf protocol implementation for Akri

Special notes for your reviewer:

I've tried (!) to avoid repeating some of the commitment feedback from http-extensibility.

If applicable:

this PR contains documentation
this PR contains unit tests
added code adheres to standard Rust formatting (cargo fmt)
code builds properly (cargo build)
code is free of common mistakes (cargo clippy)
all Akri tests succeed (cargo test)
inline documentation builds (cargo doc)
version has been updated appropriately (./version.sh)

per feedback (#162 (comment))

bfjelds · 2020-12-09T17:10:46Z

shared/src/akri/configuration.rs

+#[derive(Serialize, Deserialize, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct ZeroconfDiscoveryHandlerConfig {
+    pub filter: String,


it looks like the zeroconf crate supports several filters and this is leveraging that ... is the crate's filter syntax well known? if not, we might consider using the filter constructs that onvif and udev use (which allow inclusion and exclusion).

could be something like:

pub struct ZeroconfDiscoveryHandlerConfig { #[serde(default, skip_serializing_if = "Option::is_none")] pub names: Option<FilterList>, #[serde(default, skip_serializing_if = "Option::is_none")] pub domains: Option<FilterList>, #[serde(default, skip_serializing_if = "Option::is_none")] pub kinds: Option<FilterList>, #[serde(default, skip_serializing_if = "Option::is_none")] pub ports: Option<FilterList>, }

the crate search code could run without filter, and the discovery handler could filter.

Oh, I like this. I wish I'd thought of it. It's cleaner and clearer than what I have. I'll incorporate. Thank you.

I think much better, thank you!

Fixed: ea19332

bfjelds · 2020-12-09T17:17:53Z

samples/brokers/zeroconf/kubernetes/zeroconf.yaml

@@ -0,0 +1,18 @@
+apiVersion: akri.sh/v0


we should move this to deployment/helm/templates/zeroconf.yaml and parameterize it like the onvif/udev

bfjelds · 2020-12-11T19:08:20Z

@DazWilkin, that explanation is super helpful.

Can you quantify the bloat a little? Maybe let us know what the agent footprint is before the change and after the change (maybe for both debug and release, but most importantly for release).

I think if the bloat isn't obscene, we are happy to take it on for the moment.

In the near future, we hope to implement a solution to our monolithic problem :)

DazWilkin · 2020-12-11T19:32:26Z

Before the change is what you have now.

With zeroconf, the agent is running 829MB. It's probably that this could be improved but I'm unfamiliar with these deps.

I think perhaps, more importantly, agent users would get e.g. zeroconf regardless (as p/o the monolith) whether they want it.

And so they get any increased size but they also get its security profile and risk its CVEs.

To be clear, this is a consequence of the monolith not zeroconf but it raises the reasonable (enterprise) customer concern "Why do we need to patch the agent for a zeroconf exploit that we're not using?"

DazWilkin · 2020-12-11T19:33:13Z

I have an old v0.0.38-amd64 of the agent that is 126MB.

bfjelds · 2020-12-11T19:58:56Z

I just built locally and see that without your changes:

debug (cargo build --bin=agent): 352M
release (cargo build --bin=agent --release): 23M

can you build with --release and report how big it is?

DazWilkin · 2020-12-11T20:32:50Z

I get 21M (!?) but isn't the issue the image size?

bfjelds · 2020-12-12T15:30:42Z

I get 21M (!?) but isn't the issue the image size?

You are totally right. Sorry, the old app programmer in me hasn't fully given way to this new world yet.

DazWilkin

I get 21M (!?) but isn't the issue the image size?

You are totally right. Sorry, the old app programmer in me hasn't fully given way to this new world yet.

Nothing to apologize for :-)

bfjelds · 2020-12-14T18:33:48Z

I get 21M (!?) but isn't the issue the image size?

You are totally right. Sorry, the old app programmer in me hasn't fully given way to this new world yet.

Nothing to apologize for :-)

With a bit more time thinking, my gut says that we should hit the pause button on this until we get a different extensibility model that would allow this provider to be more on-demand. But I'll pull more people into this discussion to see if we can get more heads involved.

kate-goldenring · 2020-12-14T19:53:10Z

I get 21M (!?) but isn't the issue the image size?

You are totally right. Sorry, the old app programmer in me hasn't fully given way to this new world yet.

Nothing to apologize for :-)

With a bit more time thinking, my gut says that we should hit the pause button on this until we get a different extensibility model that would allow this provider to be more on-demand. But I'll pull more people into this discussion to see if we can get more heads involved.

I agree that this might expand the size of the Agent too much. Maybe we can do the following:

Investigate which of the Zeroconf crate dependencies is blowing up the size
Determine whether the astro-dns dependencies are smaller. @DazWilkin did you resolve the issue you were having getting astro-dns to work
If neither of the first 2 are conclusive, wait until we have an extensibility model

DazWilkin · 2020-12-14T22:24:56Z

Comment threading would be useful, nah?

The zeroconf crate stipulates dependencies (link) but, I have been able to tweak these:

Build: clang, libavahi-client-dev
Run: avahi-daemon, libavahi-client-dev

The agent's Dockerfile does not appear to be multistage but adding:

Agent: clang, libavahi-client-dev, avahi-daemon

Reduces the image size to 476MB

For a multi-stage build test, the image size was 206MB but this may be unrealistic.

I've received no response to my issue on astro-dnssd.

The issue is likely to arise with any extensions that are added to the Agent while it's a monolith. Zeroconf is likely just the first.

bfjelds · 2020-12-19T23:02:14Z

The issue is likely to arise with any extensions that are added to the Agent while it's a monolith. Zeroconf is likely just the first

No doubt. Getting a working gRPC interface into the Agent to allow out-of-proc discovery would be a valuable addition ASAP!!

kate-goldenring · 2021-03-12T21:24:24Z

Hi @DazWilkin ! The new DiscoveryHandler model has been merged. Its a big change, so I put together a document with the steps to get this PR into the new mold. Let me know how I can help -- I could put PR's into yours with the nitty stuff like a Helm template for the Configuration. Rebasing is going to be tricky, and I wonder if creating a new PR is better.

DazWilkin · 2021-03-12T22:24:55Z

Awesome work!

As you know I've had a look at your HTTP example and the gRPC-based DiscoveryHandler is great.

I think I/we should close this PR.

I'm working on another project at the moment and it's unclear when I'll get time to rewrite the Zeroconf protocol.

If I do get an opportunity, this will be top of my Akri list.

kate-goldenring · 2021-03-12T22:28:25Z

Awesome work!

As you know I've had a look at your HTTP example and the gRPC-based DiscoveryHandler is great.

I think I/we should close this PR.

I'm working on another project at the moment and it's unclear when I'll get time to rewrite the Zeroconf protocol.

If I do get an opportunity, this will be top of my Akri list.

Sounds good to me! I'll let you hit the close button @DazWilkin

DazWilkin added 23 commits November 17, 2020 12:18

ENTRYPOINT >> CMD

de6e98e

Initial commit

8a6a72b

First build

cdf75af

Revised

9c2efdd

Typo

8153dfc

Standalone implementation

fb747c8

Accompanying files

30ed316

Working

eea36d4

s/println!/trace!

2a832dd

Added zeroconf-filter

7ce53ac

Refined ZeroConf Agent Configuration

2f5e87e

Updated

77bfb82

Updated

77b642c

Add ignore file

f86043b

Add Ignore files

cff6dd8

Generalize

ee45726

Removed redundant akri-shared refs

aabe828

Add GitHub Action for ZeroConf Broker (revised MD)

b168df3

Consistency

9c2a2c9

Explain 'filter'

d5d6745

r/ZeroConf/Zeroconf

c021b76

per feedback (#162 (comment))

General filtering of services

ca5969e

Merge branch 'main' into protocol-zeroconf

0a20094

DazWilkin requested review from bfjelds, Britel, jiria and kate-goldenring as code owners December 8, 2020 22:13

DazWilkin mentioned this pull request Dec 8, 2020

[Extensibility] ZeroConf #132

Closed

bfjelds reviewed Dec 9, 2020

View reviewed changes

DazWilkin added 2 commits December 11, 2020 13:45

Move filter and add tests

e9a3ef5

Document unit tests

77238d1

DazWilkin commented Dec 12, 2020

View reviewed changes

DazWilkin added 6 commits December 15, 2020 12:34

Ensures that multiple Instances are selected

2d86080

Refactored and added unit tests

0f0530c

Include Cargo.toml change for tokio-test

75f1908

Another way to view environment variables

4fbe887

Add TXT records & unit tests

de092f7

Include TXT records in fmt::Display

e9c86b4

kate-goldenring linked an issue Jan 11, 2021 that may be closed by this pull request

[Zeroconf] WebThings are discoverable using mDNS|SD #183

Closed

kate-goldenring mentioned this pull request Jan 11, 2021

Modularize Discovery Protocols as Pods #198

Closed

kate-goldenring linked an issue Jan 11, 2021 that may be closed by this pull request

Modularize Discovery Protocols as Pods #198

Closed

DazWilkin added 3 commits January 13, 2021 09:15

Merge branch 'main' into protocol-zeroconf

4a55d66

Merge branch 'main' into protocol-zeroconf

b8efa75

Merge branch 'main' into protocol-zeroconf

c13d151

DazWilkin requested a review from romoh as a code owner February 24, 2021 19:35

DazWilkin closed this Mar 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Extensibility] Zeroconf #163

[Extensibility] Zeroconf #163

DazWilkin commented Dec 8, 2020

bfjelds Dec 9, 2020 •

edited

Loading

DazWilkin Dec 9, 2020

DazWilkin Dec 9, 2020

bfjelds Dec 9, 2020 •

edited

Loading

bfjelds commented Dec 11, 2020

DazWilkin commented Dec 11, 2020

DazWilkin commented Dec 11, 2020

bfjelds commented Dec 11, 2020

DazWilkin commented Dec 11, 2020

bfjelds commented Dec 12, 2020

DazWilkin left a comment

bfjelds commented Dec 14, 2020

kate-goldenring commented Dec 14, 2020

DazWilkin commented Dec 14, 2020

bfjelds commented Dec 19, 2020

kate-goldenring commented Mar 12, 2021

DazWilkin commented Mar 12, 2021 •

edited

Loading

kate-goldenring commented Mar 12, 2021 •

edited

Loading

[Extensibility] Zeroconf #163

[Extensibility] Zeroconf #163

Conversation

DazWilkin commented Dec 8, 2020

bfjelds Dec 9, 2020 • edited Loading

Choose a reason for hiding this comment

DazWilkin Dec 9, 2020

Choose a reason for hiding this comment

DazWilkin Dec 9, 2020

Choose a reason for hiding this comment

bfjelds Dec 9, 2020 • edited Loading

Choose a reason for hiding this comment

bfjelds commented Dec 11, 2020

DazWilkin commented Dec 11, 2020

DazWilkin commented Dec 11, 2020

bfjelds commented Dec 11, 2020

DazWilkin commented Dec 11, 2020

bfjelds commented Dec 12, 2020

DazWilkin left a comment

Choose a reason for hiding this comment

bfjelds commented Dec 14, 2020

kate-goldenring commented Dec 14, 2020

DazWilkin commented Dec 14, 2020

bfjelds commented Dec 19, 2020

kate-goldenring commented Mar 12, 2021

DazWilkin commented Mar 12, 2021 • edited Loading

kate-goldenring commented Mar 12, 2021 • edited Loading

bfjelds Dec 9, 2020 •

edited

Loading

bfjelds Dec 9, 2020 •

edited

Loading

DazWilkin commented Mar 12, 2021 •

edited

Loading

kate-goldenring commented Mar 12, 2021 •

edited

Loading