overlord,daemon,cmd: re-map snap names around the edges of snapd #5491

zyga · 2018-07-10T11:32:43Z

This branch provides the support code for allowing snapd to re-map snap names in interface interactions around the edges of the system, namely the state and the API.

The branch contains three main segments:

the remapping support code in the interface manager (two patches)
re-mapping of connection state ({get,set}Conns) (one patch)
re-mapping of API requests and responses (one patch)
discarding commented-out stale test (one patch)

Updates over weekend:

reduce the code to just snap name re-mapping
unify with and replace the nickname system
enable the core-core-system mapper (system in the API, core in state and memory)

codecov-io · 2018-07-10T18:44:53Z

Codecov Report

Merging #5491 into master will increase coverage by 0.12%.
The diff coverage is 95.09%.

@@            Coverage Diff             @@
##           master    #5491      +/-   ##
==========================================
+ Coverage   78.95%   79.07%   +0.12%     
==========================================
  Files         514      514              
  Lines       38854    38918      +64     
==========================================
+ Hits        30678    30776      +98     
+ Misses       5710     5673      -37     
- Partials     2466     2469       +3

Impacted Files	Coverage Δ
snap/info.go	`83.47% <ø> (-0.38%)`	⬇️
overlord/configstate/configstate.go	`87.5% <100%> (+2.08%)`	⬆️
daemon/api.go	`77.33% <100%> (+2.49%)`	⬆️
cmd/snap/cmd_interfaces.go	`85.45% <100%> (ø)`	⬆️
cmd/snap/cmd_interface.go	`79.54% <75%> (ø)`	⬆️
overlord/ifacestate/helpers.go	`70.91% <93.65%> (+4.46%)`	⬆️
overlord/ifacestate/handlers.go	`64.88% <0%> (-0.4%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 999c6bd...e969d39. Read the comment docs.

This patch adds re-mapping functions for plugs and slots. The idea with re-mapping is that snapd can provide the appearance of a particular plug and slot placement as observed in the on-disk state and over-the-wire communication while having a different actual placement in memory. In practice this is designed to allow us to place implicit slots on the snapd snap. Re-mapping is implemented by a golang interface. Currently only a no-op mapper is used. The mapper is global but could be moved to the interface manager after changing a large number of functions to become methods on the interface manager. This can be done in a follow-up, if desired. Apart from the golang interface, which provides the low-level operations, a number of helpers for other data types are provided. This includes connection references and plug and slot information objects. Those helpers are used by the daemon API layer in one of the subsequent patches. Two real mappers are implemented: a no-op mapper and a core<=>snapd mapper. For testing a upper<=>lower case mapper is used, for extra clarity. Everything is accompanied by unit tests. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

The helpers for re-mapping the aforementioned types are no longer necessary due to re-factoring and cleanup of the daemon code. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

This patch changes getConns and setConns to re-map connection state just after loading from the state and just before being saved to the state. This is coupled with explicit tests for {get,set}Conns that we apparently didn't have. Note that at this stage the code is still dormant since re-mapping is a no-op until the "snapd" snap becomes the host of the implicit slots. This is the "back" side of the re-mapping. The "front" side applies to re-mapping of the API layer. It is provided in the follow-up patch. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

This patch changes interface connection, disconnection and query APIs to apply re-mapping on incoming requests and outgoing responses. This allows code to issue explicit connect and disconnect API requests for the "core" snap even though implicit slots are on the "snapd" snap. Similarly the API for querying interface connections (both the legacy, as used by "snap interfaces" and the per-interface as used by "snap interface") are updated to take advantage of this feature. Note that at this stage the code is still dormant since re-mapping is a no-op until the "snapd" snap becomes the host of the implicit slots. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

We don't need a test that doesn't work. I added tests for this in the previous patch anyway. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

pedronis

the code looks good, still have some high level questions/concerns

pedronis · 2018-07-13T07:48:43Z

overlord/ifacestate/helpers.go

+// Data coming from the state and from API requests is changed so that slots on "core"
+// become slots on "snapd" (but only when "snapd" snap itself is being used). When
+// data is about to hit the state again it is re-mapped back.
+func RemapIncomingConnRef(cref *interfaces.ConnRef) (changed bool) {


I think it would be safer if these Remap* functions didn't do changes in place but where functional (the internal interface can keep being in place)

pedronis · 2018-07-13T07:48:43Z

daemon/api.go

 	for _, info := range infos {
 		// Convert interfaces.Info into interfaceJSON
 		plugs := make([]*plugJSON, 0, len(info.Plugs))
 		for _, plug := range info.Plugs {
+			// Re-map outgoing plug information.


at this point in API if understand correctly we will possibly return "core" in some situations even if the snap doesn't exist, right?

wondering if we should go all the way and as we discussed skip this intermediate state and return "system" instead, it means that the mapping to state vs the mapping for the outgoing API world need to be different, or that we need another layer of mapping just here (core => system)

Because the mapping is puggable we can easily return system here. It is all defined by the CoreSnapdMapper (which should be renamed). I can change that to return system for both snapd and core instead.

I am looking at what it would take to return "system" to the API and "core" to to the state now.

pedronis · 2018-07-13T07:48:43Z

overlord/ifacestate/helpers.go

+	return changedPlug || changedSlot
+}
+
+// RemapOutgoingPlugRef potentially re-maps the snap name of a plug reference.


given that snapd will have only implicit slots afawk, remapping of plugs is only for completeness but is not stricly needed right?

The re-mapping helper functions were returning a boolean if a mapping was actually applied (something changed) but this value is actually no longer used anywhere. In the spirit of YAGNI let's remove it. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

The remaping functions were modifying their arguments. This patch makes them return a modified copy instead. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

This patch changes a number of things, the main concept is that the mapping system can now offer separate replacements for state handling, where we want to stick to the word "core" for compatibility, and the API layer where we want to start using the "system" word as a nickname to whatever is providing implicit slots. As such the mapping functions are duplicated, once for state and once for request/response. The intermediate helpers for working with connection references are dropped because this just multiplies trivial functions that don't need to be used. The existing NilMapper is expanded to cover the new API. The existing CoreSnapdMapper is split into two new mappers: CoreSnapdSystemMapper which uses core in the state, snapd in memory and system in the API layer and a new CoreCoreSystemMapper which uses core in the state and in memory but system in the API layer. That last new mapper is a candidate for transition of spread tests that could start to talk about "system" natively (without the hacks in the client) and would then work without change once the core-snapd-system mapper can be used, which will happen once snapd starts to host implicit slots. The daemon and state are obviously updated to switch to the appopriate mapping helper. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

While testing the interface mapper I realized the nickname system is still in use and actually clashes with the interface mapper. Looking at it close it is obvious that the nickname system must be folded into the interface mapper for consistency and that the interface mapper must be able to map snap names more than than plug or slot references. Without this change we can never have implicit slots on the "snapd" snap because implicit gadget connections would resolve to "core" (which can be absent) and because the nickname handling is would clash with the idea of using "system" in the API layer but "snapd" in the memory (because nicknames would effectively change "system" to "core". To fix this I extended the interface mapper to also be a snap name mapper and removed the nickname system entirely. This also allows us to drop the ever-growing list of special cases in the client (for interface abbreviation logic). More precisely this patch: - drops the {Use,Drop}Nick functions from the snap package - changes the default mapper to CoreCoreSystem mapper - adds a SnapMapper that has request/response state save/load functions - makes InterfaceMapper a superset of SnapMapper - refactors CoreCoreSystemMapper and CoreSnapdSystemMapper - makes the daemon use RemapSnapFromRequest instead of the nick - makes gadget resolver use RemapSnapFromRequest to resolve "system" - makes the client abbreviate just "system" slots This comes without a full array of tests but without breaking any existing test (woot). It also passes spread which is promising. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

With the insight from the previous patch I have now simplified all of the interface mapper to just a SnapMapper wich can be thought as server-side nickname system. This mostly cuts the amount of boilerplate code that doesn't do anything we currently need by limiting the re-mapping to just snap names and to not be able to differentiate plugs and slots anymore. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

This patch just restores original code where it doesn't make any meaningful changes. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

mvo5

Looks nice! Thanks for working on this.

mvo5 · 2018-07-16T12:59:36Z

daemon/api_test.go

+	ifacestate.IdentityMapper // Embed the identity mapper to reuse empty state mapping functions.
+}
+
+func (m *inverseCaseMapper) RemapSnapFromRequest(snapName string) string {


(nitpick^2) I wonder if we should have something like from-req-$snapName (and vice for the RemapSnapToResponse).

mvo5 · 2018-07-16T13:57:19Z

daemon/api.go

@@ -1647,7 +1647,7 @@ func splitQS(qs string) []string {

 func getSnapConf(c *Command, r *http.Request, user *auth.UserState) Response {
 	vars := muxVars(r)
-	snapName := snap.DropNick(vars["name"])
+	snapName := ifacestate.RemapSnapFromRequest(vars["name"])


For configuration we may need our own set of of remap helpers. Last time we discussed this the idea was that we map all configuration from "system" -> "core" even on a core18 system. If this has changed thats fine but we probably still need to tweak things a bit because e.g. "canConfigure()" allows setting "core" even if core is not installed. If we change where the "system" configuration is stored we probably need to update "canConfigure"

it feels also strange to use a helper from ifacestate in configuration related code

One simple fix would be to have similar remap helpers in configstate.

This is incorrect application of IMO correct logic. This simple tweak ought to fix this:

snapName := ifacestate.RemapSnapToStore(ifacestate.RemapSnapFromRequest(vars["name"]))

pedronis

using the helpers from ifacestate for the config bits feels confusing and seems also wrong, we need to map system to core in any case there or do some other changes

pedronis · 2018-07-16T14:16:47Z

daemon/api.go

@@ -1647,7 +1647,7 @@ func splitQS(qs string) []string {

 func getSnapConf(c *Command, r *http.Request, user *auth.UserState) Response {
 	vars := muxVars(r)
-	snapName := snap.DropNick(vars["name"])
+	snapName := ifacestate.RemapSnapFromRequest(vars["name"])


it feels also strange to use a helper from ifacestate in configuration related code

pedronis · 2018-07-16T14:16:47Z

overlord/ifacestate/helpers.go

+// even if the request used "core".
+func (m *CoreSnapdSystemMapper) RemapSnapFromRequest(snapName string) string {
+	if snapName == "system" || snapName == "core" {
+		return "snapd"


this is wrong for the use we do for config!

pedronis · 2018-07-16T17:20:09Z

snap/mapper.go

+	return snapName
+}
+
+func (m *CoreSnapdSystemMapper) ConfigIfaceRemapSnapToResponse(snapName string) string {


is ConfigIface a typo? is there no tests failing?

Yeah, sorry for this.

pedronis · 2018-07-16T17:20:09Z

snap/mapper.go

+ *
+ */
+
+package snap


I'm not sure snap is the best place for this, a mapper makes sense and can be chosen only inside snapd

I pushed a simpler version that just adds a remap helper into configstate now.

pedronis

+1, with some small comments

pedronis · 2018-07-17T12:07:22Z

overlord/configstate/configstate.go

@@ -120,3 +120,19 @@ func Configure(st *state.State, snapName string, patch map[string]interface{}, f
 	task := hookstate.HookTask(st, summary, hooksup, contextData)
 	return state.NewTaskSet(task)
 }
+
+// RemapSnapFromRequest renames a snap as received from an API request
+func RemapSnapFromRequest(snapName string) string {


these need unit tests?

Yes, I just wrote those.

pedronis · 2018-07-17T12:07:22Z

overlord/ifacestate/helpers.go

+// exactly the same as with the "snapd" mapper. This can be used to make any
+// necessary adjustments the test suite.
+type CoreCoreSystemMapper struct {
+	IdentityMapper // Embedding the nil mapper allows us to cut on boilerplate.


nil mapper seems the wrong name now? same below

Fixed here and below.

pedronis · 2018-07-17T12:07:22Z

overlord/configstate/configstate.go

+}
+
+// RemapSnapToResponse renames a snap as about to be sent from an API response
+func RemapSnapToResponse(snapName string) string {


is this ever used?

No it is not. I'm looking if it should be used now.

…slot-remapping

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

pedronis · 2018-07-17T16:12:23Z

daemon/api.go

@@ -1714,7 +1714,7 @@ func setSnapConf(c *Command, r *http.Request, user *auth.UserState) Response {
 	taskset, err := configstate.ConfigureInstalled(st, snapName, patchValues, 0)
 	if err != nil {
 		if _, ok := err.(*snap.NotInstalledError); ok {
-			return SnapNotFound(snapName, err)
+			return SnapNotFound(configstate.RemapSnapToResponse(snapName), err)


we will not hit this case because core/system can be configured before anything is installed

Removed now

zyga added the ⛔ Blocked label Jul 10, 2018

zyga force-pushed the zyga:feature/plug-slot-remapping branch from 679b1de to 8056bc0 Jul 10, 2018

zyga mentioned this pull request Jul 10, 2018

overlord/ifacestate: add re-mapping function for plugs and slots #5492

Closed

zyga added 5 commits Jul 9, 2018

overlord/ifacestate: drop re-mapping snap.{Plug,Slog}Info

004aea1

The helpers for re-mapping the aforementioned types are no longer necessary due to re-factoring and cleanup of the daemon code. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

daemon: drop unused, commented-out test

Loading status checks…

72f358b

We don't need a test that doesn't work. I added tests for this in the previous patch anyway. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

zyga force-pushed the zyga:feature/plug-slot-remapping branch from 8056bc0 to 72f358b Jul 12, 2018

zyga added Core18 and removed ⛔ Blocked labels Jul 12, 2018

pedronis reviewed Jul 13, 2018

View changes

pedronis requested a review from niemeyer Jul 13, 2018

zyga added 11 commits Jul 13, 2018

overlord,daemon: make re-mapping helpers functional

Loading status checks…

dc43dd3

The remaping functions were modifying their arguments. This patch makes them return a modified copy instead. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

overlord/ifacestate: move mapper closer to related functions

8197c6b

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

overlord/ifacestate: rename NilMapper to IdentityMapper

Loading status checks…

e414445

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

overlord/ifacestate: add extra tests for better coverage

Loading status checks…

a950542

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

overlord/ifacestate: remove uneeded teardown method

df33bd3

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

daemon: avoid changing existing ConnRef

039be9e

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

daemon: undo needless patch delta

Loading status checks…

360ad8e

This patch just restores original code where it doesn't make any meaningful changes. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

zyga changed the title ~~overlord,daemon: re-map interface plugs and slots around the edges of snapd~~ overlord,daemon,cmd: re-map snap names around the edges of snapd Jul 13, 2018

mvo5 reviewed Jul 16, 2018

View changes

pedronis requested changes Jul 16, 2018

View changes

configstate: use config specific mapper

Loading status checks…

292c38b

pedronis reviewed Jul 16, 2018

View changes

mvo5 force-pushed the zyga:feature/plug-slot-remapping branch from ff338c7 to 292c38b Jul 16, 2018

pedronis approved these changes Jul 17, 2018

View changes

zyga added 3 commits Jul 17, 2018

Merge branch 'master' of github.com:snapcore/snapd into feature/plug-…

61e4d18

…slot-remapping

overlord/ifacestate: update comments to refer to identity mapper

4f71891

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

overlord/confistate: add tests for remappers

Loading status checks…

339a091

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

pedronis reviewed Jul 17, 2018

View changes

mvo5 approved these changes Jul 17, 2018

View changes

zyga force-pushed the zyga:feature/plug-slot-remapping branch from e969d39 to 339a091 Jul 17, 2018

mvo5 merged commit 1544ced into snapcore:master Jul 17, 2018
1 check passed

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

zyga deleted the zyga:feature/plug-slot-remapping branch Jul 17, 2018

snapcore / snapd

overlord,daemon,cmd: re-map snap names around the edges of snapd #5491

overlord,daemon,cmd: re-map snap names around the edges of snapd #5491

zyga commented Jul 10, 2018 •

edited

codecov-io commented Jul 10, 2018 •

edited

pedronis left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

mvo5 left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

pedronis left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

pedronis left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

snapcore / snapd

overlord,daemon,cmd: re-map snap names around the edges of snapd #5491

overlord,daemon,cmd: re-map snap names around the edges of snapd #5491

Conversation

zyga commented Jul 10, 2018 • edited

codecov-io commented Jul 10, 2018 • edited

Codecov Report

pedronis left a comment

This comment has been minimized.

pedronis Jul 13, 2018 Collaborator

This comment has been minimized.

zyga Jul 13, 2018 Author Collaborator

This comment has been minimized.

pedronis Jul 13, 2018 Collaborator

This comment has been minimized.

zyga Jul 13, 2018 Author Collaborator

This comment has been minimized.

zyga Jul 13, 2018 Author Collaborator

This comment has been minimized.

pedronis Jul 13, 2018 Collaborator

This comment has been minimized.

zyga Jul 13, 2018 Author Collaborator

mvo5 left a comment

This comment has been minimized.

mvo5 Jul 16, 2018 Collaborator

This comment has been minimized.

mvo5 Jul 16, 2018 Collaborator

This comment has been minimized.

pedronis Jul 16, 2018 Collaborator

This comment has been minimized.

mvo5 Jul 16, 2018 Collaborator

This comment has been minimized.

zyga Jul 16, 2018 Author Collaborator

pedronis left a comment

This comment has been minimized.

pedronis Jul 16, 2018 Collaborator

This comment has been minimized.

pedronis Jul 16, 2018 Collaborator

This comment has been minimized.

pedronis Jul 16, 2018 Collaborator

This comment has been minimized.

mvo5 Jul 16, 2018 Collaborator

This comment has been minimized.

pedronis Jul 16, 2018 Collaborator

This comment has been minimized.

mvo5 Jul 16, 2018 Collaborator

pedronis left a comment

This comment has been minimized.

pedronis Jul 17, 2018 Collaborator

This comment has been minimized.

zyga Jul 17, 2018 Author Collaborator

This comment has been minimized.

pedronis Jul 17, 2018 Collaborator

This comment has been minimized.

zyga Jul 17, 2018 Author Collaborator

This comment has been minimized.

pedronis Jul 17, 2018 Collaborator

This comment has been minimized.

zyga Jul 17, 2018 Author Collaborator

This comment has been minimized.

zyga Jul 17, 2018 Author Collaborator

This comment has been minimized.

pedronis Jul 17, 2018 Collaborator

This comment has been minimized.

zyga Jul 17, 2018 Author Collaborator

zyga commented Jul 10, 2018 •

edited

codecov-io commented Jul 10, 2018 •

edited

pedronis Jul 13, 2018
Collaborator

zyga Jul 13, 2018
Author Collaborator

pedronis Jul 13, 2018
Collaborator

zyga Jul 13, 2018
Author Collaborator

zyga Jul 13, 2018
Author Collaborator

pedronis Jul 13, 2018
Collaborator

zyga Jul 13, 2018
Author Collaborator

mvo5 Jul 16, 2018
Collaborator

mvo5 Jul 16, 2018
Collaborator

pedronis Jul 16, 2018
Collaborator

mvo5 Jul 16, 2018
Collaborator

zyga Jul 16, 2018
Author Collaborator

pedronis Jul 16, 2018
Collaborator

pedronis Jul 16, 2018
Collaborator

pedronis Jul 16, 2018
Collaborator

mvo5 Jul 16, 2018
Collaborator

pedronis Jul 16, 2018
Collaborator

mvo5 Jul 16, 2018
Collaborator

pedronis Jul 17, 2018
Collaborator

zyga Jul 17, 2018
Author Collaborator

pedronis Jul 17, 2018
Collaborator

zyga Jul 17, 2018
Author Collaborator

pedronis Jul 17, 2018
Collaborator

zyga Jul 17, 2018
Author Collaborator

zyga Jul 17, 2018
Author Collaborator

pedronis Jul 17, 2018
Collaborator

zyga Jul 17, 2018
Author Collaborator