New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ENI Plugin Implementation #1
Conversation
The logic of spec compatibility is moved to this package from a hardcoded string in the main package
| "github.com/stretchr/testify/assert" | ||
| ) | ||
|
|
||
| func TestGetSpecVersionsSupported(t *testing.T) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the test shouldn't depend on the value in the main package which may change in the future.
If you want to test the GetSpecVersionsSupported, you should assign specVersionSupported some random value and then test against the value. Please correct me if I'm misunderstood.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed, will modify this.
| // * VERSION | ||
| // Refer to https://github.com/containernetworking/cni/blob/master/SPEC.md | ||
| // for details | ||
| var specVersionSupported = version.PluginSupports("0.2.0") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not start from 0.0.0?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As per the description, this plugin supports add, del and version commands, which means that it's not compatible with anything less than that. 0.3.0 would mean supporting additional things, which is why this is set to 0.2.0 for now.
plugins/eni/types/types.go
Outdated
| return nil, errors.Wrapf(err, "Failed to parse config") | ||
| } | ||
| if conf.ENIID == "" { | ||
| return nil, fmt.Errorf("Missing required parameter in config: '%s'", "eni") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since you are using errors package, you can also use errors here: errors.Errorf()
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, will modify this.
plugins/eni/types/types.go
Outdated
| if conf.ENIID == "" { | ||
| return nil, fmt.Errorf("Missing required parameter in config: '%s'", "eni") | ||
| } | ||
| if conf.IPV4Address == "" { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should also verify the format of the address.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fair enough
plugins/eni/engine/engine.go
Outdated
| var macAddresses []string | ||
| macs, err := engine.metadata.GetMetadata(metadataNetworkInterfacesPath) | ||
| if err != nil { | ||
| return macAddresses, errors.Wrapf(err, "Error getting all mac addresses on the instance from instance metadata") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
errors.Wrap? Also won't nil work here for macAddresses?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would, i can change it.
plugins/eni/engine/engine.go
Outdated
| } | ||
| } | ||
|
|
||
| return "", fmt.Errorf("Network device name not found for '%s'", macAddress) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
errors.Errorf?
plugins/eni/engine/engine.go
Outdated
| func (engine *engine) GetInterfaceDeviceName(macAddress string) (string, error) { | ||
| files, err := engine.ioutil.ReadDir(sysfsPathForNetworkDevices) | ||
| if err != nil { | ||
| return "", errors.Wrapf(err, "Error listing network devices from sys fs") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
errors.Wrap?
| const ( | ||
| metadataNetworkInterfacesPath = "network/interfaces/macs/" | ||
| metadataNetworkInterfaceIDPathSuffix = "interface-id" | ||
| sysfsPathForNetworkDevices = "/sys/class/net/" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will this stays the same on ubuntu/coreOS?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i've verified that it's the same on ubuntu. @nmeyerhans thinks that it should be the same path as well on other distributions as it's the sysfs mount and should remain consistent.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's probably the same for most distributions. Let's leave it as-is for now and change it if we discover a distribution that is different.
plugins/eni/engine/engine.go
Outdated
| } | ||
|
|
||
| parts = strings.Split(cidr, ".") | ||
| if len(parts) != 4 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can use a regex to match the ip, like: [0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can do better. We can just parse it using the ParseAddr api from netlink
| @@ -0,0 +1,18 @@ | |||
| SOURCEDIR=./pkg ./plugins | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When do we include the dependency inside this repo?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what do you mean?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I mean put the two dependencies in the vendor folder, so that it can be built?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm adding dependencies in a separate PR. Otherwise, it'll make this unreadable.
…ith methods from the net package
plugins/eni/types/types_test.go
Outdated
| StdinData: []byte(configInvalidIPV4AddressIPv6), | ||
| } | ||
| _, err := NewConf(args) | ||
| t.Log(err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove this line?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it slipped in somehow.
scripts/mockgen.go
Outdated
| path, _ := filepath.Split(outputPath) | ||
| err := os.MkdirAll(path, os.ModeDir|0755) | ||
| if err != nil { | ||
| fmt.Println(err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we wrapper this two line in a function, since it occurs multiple times?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alright
| if conf.ENIID == "" { | ||
| return nil, errors.Errorf("Missing required parameter in config: '%s'", "eni") | ||
| } | ||
| if conf.IPV4Address == "" { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can this check combine with the next if?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll add more text into the error message. I want to keep this separate so that we can tell explicitly which parameter is missing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looked at the code again. It does print generate different error messages. I'll leave it as is.
| go get golang.org/x/tools/cmd/goimports | ||
| go get github.com/tools/godep | ||
|
|
||
| .PHONY: plugins |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you make a non-PHONY target that corresponds to the generated binary? See $(LOCAL_BINARY) in https://github.com/awslabs/amazon-ecr-credential-helper/blob/master/Makefile for an example.
plugins/eni/commands/commands.go
Outdated
| ec2metadata.NewEC2Metadata(), ioutilwrapper.NewIOUtil(), netlinkwrapper.NewNetLink(), cninswrapper.NewNS())) | ||
| } | ||
|
|
||
| // Del invokes the command to remove ENI to a container's namespace |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: should be "from a container's namespace"
| // express or implied. See the License for the specific language governing | ||
| // permissions and limitations under the License. | ||
|
|
||
| package engine |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we have a different package name? Not that I have any specific suggestions, just curious.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Couldn't think of a better name either. How does workflow sound compared to this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that's somewhat better. Still going to see if I can come up with other suggestions too.
plugins/eni/commands/commands.go
Outdated
| if !ok { | ||
| errorMessage := "Unable to map ipv4 address of ENI to a mac address" | ||
| log.Error(errorMessage) | ||
| return errors.Errorf(errorMessage) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
errors should begin with lower-case and some contextual information. Maybe "add: " + errorMessage.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I read https://golang.org/doc/effective_go.html#errors this morning. I'm mortified that I've written so many bad error messages till now (especially in terms of lacking source and package information). Will rectify this!
When feasible, error strings should identify their origin, such as by having a prefix naming the operation or package that generated the error. For example, in package image, the string representation for a decoding error due to an unknown format is "image: unknown format".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Me too. I became more aware of this recently at the Seattle Go Programmers meetup/Go 1.8 release party.
| const ( | ||
| metadataNetworkInterfacesPath = "network/interfaces/macs/" | ||
| metadataNetworkInterfaceIDPathSuffix = "interface-id" | ||
| sysfsPathForNetworkDevices = "/sys/class/net/" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's probably the same for most distributions. Let's leave it as-is for now and change it if we discover a distribution that is different.
plugins/eni/engine/engine.go
Outdated
| } | ||
|
|
||
| // NewEngine creates a new Engine object | ||
| func NewEngine(metadata ec2metadata.EC2Metadata, ioutil ioutilwrapper.IOUtil, netLink netlinkwrapper.NetLink, ns cninswrapper.NS) Engine { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: engine.New to avoid the stutter
plugins/eni/engine/engine.go
Outdated
|
|
||
| // NewEngine creates a new Engine object | ||
| func NewEngine(metadata ec2metadata.EC2Metadata, ioutil ioutilwrapper.IOUtil, netLink netlinkwrapper.NetLink, ns cninswrapper.NS) Engine { | ||
| return &engine{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should New just create the wrappers instead of having them passed in? Are you using this for more than just unit tests of the engine package?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll add another level of indirection here.
Made New() a wrapper that creates the necessary fields for the 'engine' obect. Also modified the 'Makefile' so that the eni executable is built as a non PHONY target
Added IsDHClientInPath() method to the Engine interface. Updated engine create() method and tests accordingly
…h ENI Modify namespace closure execution to invoke dhclient to start renewing the lease on the IPV4 address of the ENI
plugins/eni/engine/engine.go
Outdated
| @@ -56,7 +56,12 @@ type engine struct { | |||
| } | |||
|
|
|||
| // NewEngine creates a new Engine object | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: New
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lgtm, just minor comments.
plugins/eni/commands/commands.go
Outdated
| @@ -38,6 +45,11 @@ func add(args *skel.CmdArgs, engine engine.Engine) error { | |||
| return err | |||
| } | |||
|
|
|||
| if ok := engine.IsDHClientInPath(); !ok { | |||
| log.Errorf("Unable to fing the dhclient executable") | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: log.Error("Unable to find the dhclient executable")
plugins/eni/commands/commands.go
Outdated
| @@ -21,6 +21,13 @@ import ( | |||
| "github.com/pkg/errors" | |||
| ) | |||
|
|
|||
| var ( | |||
| unmappedIPV4AddressError = errors.Errorf( | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not just reuse the newUnmappedIPV4AddressError in engine/errors. Also it should be errors.New.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a type and this is a variable, both intended to be used for different things. Does it matter if it's errors.New() or errors.Errorf()? I'm not sure I understand the difference between the two in terms of creating error objects from strings.
plugins/eni/commands/commands.go
Outdated
| err = newUnmappedIPV4AddressError("add", "commands", "unable to map ipv4 address of ENI to a mac address") | ||
| log.Error("Unable to validate ipv4 address for ENI: %v", err) | ||
| return err | ||
| log.Error("Unable to validate ipv4 address for ENI: %v", unmappedIPV4AddressError) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Errorf
| // expect the Agent to create those directories. We can also let | ||
| // these be conigured via the plugin config. | ||
| dhclientV4LeaseFilePathPrefix = "/var/lib/dhclient/ns-dhclient" | ||
| dhclientV4LeasePIDFilePathPrefix = "/var/run/ns-dhclient" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are these different in container namespace, as I get /var/run/dhclient-xxx on the host?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The mount namespace is same as where it's being invoked from. If the plugin is invoked on the host using scripts in cni package, it'd be the mount namespace of the host. If the plugin is invoked from the ECS Agent container, it'd be the mount namespace of the agent.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we planning to invoke this from the ECS agent container?
|
@richardpen Thanks for reviewing this. I'll address your comments in a new PR as I want to merge this today. I don't want this one to drag on as I'm continuing to make more changes. @vsiddharth @samuelkarp Feel free to add comments in this PR. I'll address them in a separate PR. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good. I have a bunch of nitpicks, but none of these are critical to address immediately.
| go generate -x ./pkg/... ./plugins/... | ||
|
|
||
| .PHONY: unit-tests | ||
| unit-tests: $(SOURCES) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: I would probably prefer test here since I find it slightly more natural to type make test. But this is not a big deal.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We'll get this when integrate #3 in.
| ) | ||
|
|
||
| var ( | ||
| unmappedIPV4AddressError = errors.Errorf( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: These should be errors.New since there is no format string here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
|
|
||
| var ( | ||
| unmappedIPV4AddressError = errors.Errorf( | ||
| "add commands: unable to map ipv4 address of ENI to a mac address") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: I think I'd prefer starting these with "eni add" since that encapsulates the plugin name as well as the operation and I assume these errors become visible outside the plugin.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we use different log locations for different plugins, we might not need this. We can come back to this.
| // TODO: If we can get this information from the config, we can optimize | ||
| // the workflow by getting rid of this, or by making this optional (only | ||
| // in cases where mac address hasn't been specified) | ||
| allMACAddresses, err := engine.GetAllMACAddresses() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: I think we're mixing abstraction layers here and it would be nice to get them separate early on. In order to add, you need to obtain the requisite information and then call engine.SetupContainerNamespace, and in order to call that you need the MAC, device name, and subnet mask. It seems trivial, but I think it'd be good to move the MAC resolution + validation out to a separate function so that add just looks like it gathers the necessary arguments to setup the namespace and then calls in to actually set it up.
func getMACforENI(eniID string, ipv4address string, engine engine.Engine) (string,nil) {
allMACAddresses, err := engine.GetAllMACAddresses()
//...
macAddressOfENI, err := engine.GetMACAddressOfENI(allMACAddresses, eniID)
//...
ok, err := engine.DoesMACAddressMapToIPV4Address(macAddressOfENI, ipv4Address)
//...
return macAddressOfENI, nil
}There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
| // expect the Agent to create those directories. We can also let | ||
| // these be conigured via the plugin config. | ||
| dhclientV4LeaseFilePathPrefix = "/var/lib/dhclient/ns-dhclient" | ||
| dhclientV4LeasePIDFilePathPrefix = "/var/run/ns-dhclient" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we planning to invoke this from the ECS agent container?
| return false | ||
| } | ||
|
|
||
| log.Debugf("dhclient found in: %s", dhclientPath) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: we might want to change this to Tracef in the future if debug gets too noisy
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we planning to invoke this from the ECS agent container?
This is invoked as a part of setup namespace closure: https://github.com/aws/amazon-ecs-cni-plugins/blob/dev/plugins/eni/engine/nsclosure.go#L115
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm trying to understand more about your response to @richardpen above:
The mount namespace is same as where it's being invoked from. If the plugin is invoked on the host using scripts in cni package, it'd be the mount namespace of the host. If the plugin is invoked from the ECS Agent container, it'd be the mount namespace of the agent.
Are we invoking it on the host using scripts in the cni package or from the ECS agent container?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, that would be from the ECS Agent. We'd have mount the executable from the host to the agent's mount namespace as well.
| macs, err := engine.metadata.GetMetadata(metadataNetworkInterfacesPath) | ||
| if err != nil { | ||
| return nil, errors.Wrap(err, | ||
| "getAllMACAddresses engine: unable to get all mac addresses on the instance from instance metadata") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(multiple occurrences) nit: I think it's safe to just start this with "engine" as the rest of the error string is unique enough to find it in source.
| return nil | ||
| } | ||
|
|
||
| // nsClosure wraps the parameters and the method to configure the container's namespace |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: I wonder if this might be better named as nsContext?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I got the name from https://github.com/containernetworking/cni/blob/94e02e0768ffeeb5c4eec57184b3bc8d11650e37/pkg/ns/ns.go#L169. It's really context for executing the closure. I'll rename it to reflect that.
| // cannot be mapped to any of the network interfaces attached to the host as | ||
| // determined by the instance metadata | ||
| type unmappedMACAddressError struct { | ||
| err *_error |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(multiple occurrences) Any reason not to just embed _error here? That would free you from having to make a new Error method and deal with an extra pointer.
Summary
The ENI plugin assigns the ENI to a container's namespace and sets it up as per it's subnet configuration parameters. The
Add()operation sets up the ENI in a container's namespace.Del()tears it down. This PR implementsAdd().Implementation Details
pkgmostly contains interfaces around various dependent packages to ease testingplugins/enicontains the implementation of the ENI pluginplugins/eni/commandscontains the implementation ofAddandDelcommandsplugins/eni/enginecontains the implementation of the execution engine of the pluginTesting Done