overlord,daemon,cmd: re-map snap names around the edges of snapd #5491
Conversation
Codecov Report
@@ Coverage Diff @@
## master #5491 +/- ##
==========================================
+ Coverage 78.95% 79.07% +0.12%
==========================================
Files 514 514
Lines 38854 38918 +64
==========================================
+ Hits 30678 30776 +98
+ Misses 5710 5673 -37
- Partials 2466 2469 +3
Continue to review full report at Codecov.
|
This patch adds re-mapping functions for plugs and slots. The idea with re-mapping is that snapd can provide the appearance of a particular plug and slot placement as observed in the on-disk state and over-the-wire communication while having a different actual placement in memory. In practice this is designed to allow us to place implicit slots on the snapd snap. Re-mapping is implemented by a golang interface. Currently only a no-op mapper is used. The mapper is global but could be moved to the interface manager after changing a large number of functions to become methods on the interface manager. This can be done in a follow-up, if desired. Apart from the golang interface, which provides the low-level operations, a number of helpers for other data types are provided. This includes connection references and plug and slot information objects. Those helpers are used by the daemon API layer in one of the subsequent patches. Two real mappers are implemented: a no-op mapper and a core<=>snapd mapper. For testing a upper<=>lower case mapper is used, for extra clarity. Everything is accompanied by unit tests. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
The helpers for re-mapping the aforementioned types are no longer necessary due to re-factoring and cleanup of the daemon code. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
This patch changes getConns and setConns to re-map connection state just after loading from the state and just before being saved to the state. This is coupled with explicit tests for {get,set}Conns that we apparently didn't have. Note that at this stage the code is still dormant since re-mapping is a no-op until the "snapd" snap becomes the host of the implicit slots. This is the "back" side of the re-mapping. The "front" side applies to re-mapping of the API layer. It is provided in the follow-up patch. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
This patch changes interface connection, disconnection and query APIs to apply re-mapping on incoming requests and outgoing responses. This allows code to issue explicit connect and disconnect API requests for the "core" snap even though implicit slots are on the "snapd" snap. Similarly the API for querying interface connections (both the legacy, as used by "snap interfaces" and the per-interface as used by "snap interface") are updated to take advantage of this feature. Note that at this stage the code is still dormant since re-mapping is a no-op until the "snapd" snap becomes the host of the implicit slots. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
We don't need a test that doesn't work. I added tests for this in the previous patch anyway. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
the code looks good, still have some high level questions/concerns |
// Data coming from the state and from API requests is changed so that slots on "core" | ||
// become slots on "snapd" (but only when "snapd" snap itself is being used). When | ||
// data is about to hit the state again it is re-mapped back. | ||
func RemapIncomingConnRef(cref *interfaces.ConnRef) (changed bool) { |
pedronis
Jul 13, 2018
Collaborator
I think it would be safer if these Remap* functions didn't do changes in place but where functional (the internal interface can keep being in place)
I think it would be safer if these Remap* functions didn't do changes in place but where functional (the internal interface can keep being in place)
zyga
Jul 13, 2018
Author
Collaborator
Done
Done
for _, info := range infos { | ||
// Convert interfaces.Info into interfaceJSON | ||
plugs := make([]*plugJSON, 0, len(info.Plugs)) | ||
for _, plug := range info.Plugs { | ||
// Re-map outgoing plug information. |
pedronis
Jul 13, 2018
Collaborator
at this point in API if understand correctly we will possibly return "core" in some situations even if the snap doesn't exist, right?
wondering if we should go all the way and as we discussed skip this intermediate state and return "system" instead, it means that the mapping to state vs the mapping for the outgoing API world need to be different, or that we need another layer of mapping just here (core => system)
at this point in API if understand correctly we will possibly return "core" in some situations even if the snap doesn't exist, right?
wondering if we should go all the way and as we discussed skip this intermediate state and return "system" instead, it means that the mapping to state vs the mapping for the outgoing API world need to be different, or that we need another layer of mapping just here (core => system)
zyga
Jul 13, 2018
Author
Collaborator
Because the mapping is puggable we can easily return system here. It is all defined by the CoreSnapdMapper (which should be renamed). I can change that to return system for both snapd and core instead.
Because the mapping is puggable we can easily return system here. It is all defined by the CoreSnapdMapper (which should be renamed). I can change that to return system for both snapd and core instead.
zyga
Jul 13, 2018
Author
Collaborator
I am looking at what it would take to return "system" to the API and "core" to to the state now.
I am looking at what it would take to return "system" to the API and "core" to to the state now.
return changedPlug || changedSlot | ||
} | ||
|
||
// RemapOutgoingPlugRef potentially re-maps the snap name of a plug reference. |
pedronis
Jul 13, 2018
Collaborator
given that snapd will have only implicit slots afawk, remapping of plugs is only for completeness but is not stricly needed right?
given that snapd will have only implicit slots afawk, remapping of plugs is only for completeness but is not stricly needed right?
zyga
Jul 13, 2018
Author
Collaborator
Exactly so
Exactly so
The re-mapping helper functions were returning a boolean if a mapping was actually applied (something changed) but this value is actually no longer used anywhere. In the spirit of YAGNI let's remove it. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
The remaping functions were modifying their arguments. This patch makes them return a modified copy instead. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
This patch changes a number of things, the main concept is that the mapping system can now offer separate replacements for state handling, where we want to stick to the word "core" for compatibility, and the API layer where we want to start using the "system" word as a nickname to whatever is providing implicit slots. As such the mapping functions are duplicated, once for state and once for request/response. The intermediate helpers for working with connection references are dropped because this just multiplies trivial functions that don't need to be used. The existing NilMapper is expanded to cover the new API. The existing CoreSnapdMapper is split into two new mappers: CoreSnapdSystemMapper which uses core in the state, snapd in memory and system in the API layer and a new CoreCoreSystemMapper which uses core in the state and in memory but system in the API layer. That last new mapper is a candidate for transition of spread tests that could start to talk about "system" natively (without the hacks in the client) and would then work without change once the core-snapd-system mapper can be used, which will happen once snapd starts to host implicit slots. The daemon and state are obviously updated to switch to the appopriate mapping helper. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
While testing the interface mapper I realized the nickname system is still in use and actually clashes with the interface mapper. Looking at it close it is obvious that the nickname system must be folded into the interface mapper for consistency and that the interface mapper must be able to map snap names more than than plug or slot references. Without this change we can never have implicit slots on the "snapd" snap because implicit gadget connections would resolve to "core" (which can be absent) and because the nickname handling is would clash with the idea of using "system" in the API layer but "snapd" in the memory (because nicknames would effectively change "system" to "core". To fix this I extended the interface mapper to also be a snap name mapper and removed the nickname system entirely. This also allows us to drop the ever-growing list of special cases in the client (for interface abbreviation logic). More precisely this patch: - drops the {Use,Drop}Nick functions from the snap package - changes the default mapper to CoreCoreSystem mapper - adds a SnapMapper that has request/response state save/load functions - makes InterfaceMapper a superset of SnapMapper - refactors CoreCoreSystemMapper and CoreSnapdSystemMapper - makes the daemon use RemapSnapFromRequest instead of the nick - makes gadget resolver use RemapSnapFromRequest to resolve "system" - makes the client abbreviate just "system" slots This comes without a full array of tests but without breaking any existing test (woot). It also passes spread which is promising. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
With the insight from the previous patch I have now simplified all of the interface mapper to just a SnapMapper wich can be thought as server-side nickname system. This mostly cuts the amount of boilerplate code that doesn't do anything we currently need by limiting the re-mapping to just snap names and to not be able to differentiate plugs and slots anymore. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
This patch just restores original code where it doesn't make any meaningful changes. Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Looks nice! Thanks for working on this. |
ifacestate.IdentityMapper // Embed the identity mapper to reuse empty state mapping functions. | ||
} | ||
|
||
func (m *inverseCaseMapper) RemapSnapFromRequest(snapName string) string { |
mvo5
Jul 16, 2018
Collaborator
(nitpick^2) I wonder if we should have something like from-req-$snapName
(and vice for the RemapSnapToResponse).
(nitpick^2) I wonder if we should have something like from-req-$snapName
(and vice for the RemapSnapToResponse).
@@ -1647,7 +1647,7 @@ func splitQS(qs string) []string { | |||
|
|||
func getSnapConf(c *Command, r *http.Request, user *auth.UserState) Response { | |||
vars := muxVars(r) | |||
snapName := snap.DropNick(vars["name"]) | |||
snapName := ifacestate.RemapSnapFromRequest(vars["name"]) |
mvo5
Jul 16, 2018
Collaborator
For configuration we may need our own set of of remap helpers. Last time we discussed this the idea was that we map all configuration from "system" -> "core" even on a core18 system. If this has changed thats fine but we probably still need to tweak things a bit because e.g. "canConfigure()" allows setting "core" even if core is not installed. If we change where the "system" configuration is stored we probably need to update "canConfigure"
For configuration we may need our own set of of remap helpers. Last time we discussed this the idea was that we map all configuration from "system" -> "core" even on a core18 system. If this has changed thats fine but we probably still need to tweak things a bit because e.g. "canConfigure()" allows setting "core" even if core is not installed. If we change where the "system" configuration is stored we probably need to update "canConfigure"
pedronis
Jul 16, 2018
Collaborator
it feels also strange to use a helper from ifacestate in configuration related code
it feels also strange to use a helper from ifacestate in configuration related code
mvo5
Jul 16, 2018
Collaborator
One simple fix would be to have similar remap helpers in configstate.
One simple fix would be to have similar remap helpers in configstate.
zyga
Jul 16, 2018
Author
Collaborator
This is incorrect application of IMO correct logic. This simple tweak ought to fix this:
snapName := ifacestate.RemapSnapToStore(ifacestate.RemapSnapFromRequest(vars["name"]))
This is incorrect application of IMO correct logic. This simple tweak ought to fix this:
snapName := ifacestate.RemapSnapToStore(ifacestate.RemapSnapFromRequest(vars["name"]))
using the helpers from ifacestate for the config bits feels confusing and seems also wrong, we need to map system to core in any case there or do some other changes |
@@ -1647,7 +1647,7 @@ func splitQS(qs string) []string { | |||
|
|||
func getSnapConf(c *Command, r *http.Request, user *auth.UserState) Response { | |||
vars := muxVars(r) | |||
snapName := snap.DropNick(vars["name"]) | |||
snapName := ifacestate.RemapSnapFromRequest(vars["name"]) |
pedronis
Jul 16, 2018
Collaborator
it feels also strange to use a helper from ifacestate in configuration related code
it feels also strange to use a helper from ifacestate in configuration related code
// even if the request used "core". | ||
func (m *CoreSnapdSystemMapper) RemapSnapFromRequest(snapName string) string { | ||
if snapName == "system" || snapName == "core" { | ||
return "snapd" |
pedronis
Jul 16, 2018
Collaborator
this is wrong for the use we do for config!
this is wrong for the use we do for config!
return snapName | ||
} | ||
|
||
func (m *CoreSnapdSystemMapper) ConfigIfaceRemapSnapToResponse(snapName string) string { |
pedronis
Jul 16, 2018
Collaborator
is ConfigIface a typo? is there no tests failing?
is ConfigIface a typo? is there no tests failing?
mvo5
Jul 16, 2018
Collaborator
Yeah, sorry for this.
Yeah, sorry for this.
* | ||
*/ | ||
|
||
package snap |
pedronis
Jul 16, 2018
Collaborator
I'm not sure snap is the best place for this, a mapper makes sense and can be chosen only inside snapd
I'm not sure snap is the best place for this, a mapper makes sense and can be chosen only inside snapd
mvo5
Jul 16, 2018
Collaborator
I pushed a simpler version that just adds a remap helper into configstate now.
I pushed a simpler version that just adds a remap helper into configstate now.
+1, with some small comments |
@@ -120,3 +120,19 @@ func Configure(st *state.State, snapName string, patch map[string]interface{}, f | |||
task := hookstate.HookTask(st, summary, hooksup, contextData) | |||
return state.NewTaskSet(task) | |||
} | |||
|
|||
// RemapSnapFromRequest renames a snap as received from an API request | |||
func RemapSnapFromRequest(snapName string) string { |
pedronis
Jul 17, 2018
Collaborator
these need unit tests?
these need unit tests?
zyga
Jul 17, 2018
Author
Collaborator
Yes, I just wrote those.
Yes, I just wrote those.
// exactly the same as with the "snapd" mapper. This can be used to make any | ||
// necessary adjustments the test suite. | ||
type CoreCoreSystemMapper struct { | ||
IdentityMapper // Embedding the nil mapper allows us to cut on boilerplate. |
pedronis
Jul 17, 2018
Collaborator
nil mapper seems the wrong name now? same below
nil mapper seems the wrong name now? same below
zyga
Jul 17, 2018
Author
Collaborator
Fixed here and below.
Fixed here and below.
} | ||
|
||
// RemapSnapToResponse renames a snap as about to be sent from an API response | ||
func RemapSnapToResponse(snapName string) string { |
pedronis
Jul 17, 2018
Collaborator
is this ever used?
is this ever used?
zyga
Jul 17, 2018
Author
Collaborator
No it is not. I'm looking if it should be used now.
No it is not. I'm looking if it should be used now.
zyga
Jul 17, 2018
Author
Collaborator
Done now
Done now
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
@@ -1714,7 +1714,7 @@ func setSnapConf(c *Command, r *http.Request, user *auth.UserState) Response { | |||
taskset, err := configstate.ConfigureInstalled(st, snapName, patchValues, 0) | |||
if err != nil { | |||
if _, ok := err.(*snap.NotInstalledError); ok { | |||
return SnapNotFound(snapName, err) | |||
return SnapNotFound(configstate.RemapSnapToResponse(snapName), err) |
pedronis
Jul 17, 2018
Collaborator
we will not hit this case because core/system can be configured before anything is installed
we will not hit this case because core/system can be configured before anything is installed
zyga
Jul 17, 2018
Author
Collaborator
Removed now
Removed now
This branch provides the support code for allowing snapd to re-map snap names in interface interactions around the edges of the system, namely the state and the API.
The branch contains three main segments:
{get,set}Conns
) (one patch)Updates over weekend:
core-core-system
mapper (system in the API, core in state and memory)