V2 types and TPR #266

soamvasani · 2017-07-26T16:04:49Z

This changes the core fission function, environment and trigger types. It also changes Fission's storage to use ThirdPartyResources.

Functions are now specified by packages. Functions can also have both source and deployment packages. A package can be specified by a literal, or by a URL.
Environments have a build and runtime component.
Triggers reference functions by a FunctionReference. This is a layer of indirection between triggers and functions, and will allow things like incremental function upgrades in future releases.

See Documentation/wip/env-v2.md for design discussion about points 1 and 2.

Operationally, moving our state to TPRs means that users no longer have to maintain an etcd instance just for fission, and they also don't have to use persistent volumes for the fission api server. It also means they can specify functions and triggers declaratively and version control these declarations.

This change breaks compatibility with the v1 Fission API. It removes versioning from the core API. Some points of rationale for this:

Moving our type system closer to Kubernetes architectural patterns is important, gives us declarative infrastructure, allows better interoperability with other K8s projects, and the potential to use tools like Helm to install functions in the future.
Versioning in the v1 Fission API was based on immutability of objects. It also had a very implicit notion of "latest". Both these concepts don't fit cleanly into K8s API patterns.
We also wanted to avoid doing too much in one layer; it is best to look at the core of fission as a platform that you deploy functions into, and implement some sort of "Versioned Function" object on top of this layer.
Adding a v1 compatibility layer was in theory possible, but messy because of the immutability in v1. At the current pre-beta stage of the project it wasn't worthwhile adding and maintaining this complexity.
Broadly, we take breaking compatibility very seriously, but we're early enough in the project to be able to do it.

Note that all v1 environments are still compatible Fission even after the API change. This is largely because environments don't have fission-api-specific stuff in them.

The fission CLI retains all of its old commands, but loses the versioning semantics. To migrate from a v1 to v2 deployment, you have to use the v1 CLI to get functions, envs and triggers; and then use the v2 CLI to create them again. This is a rather manual process. We'll be happy to help anyone having trouble with this process on the fission slack channel.

This is not a complete Environment v2 implementation; the remaining changes for that are:

adding zip file support to fetcher
implementing a storage service for large packages
build management: a build manager to invoke builders, and API and CLI support for running builds, showing errors, etc.

soamvasani · 2017-07-26T16:08:27Z

cc @yqf3139 @dgoujard This change breaks compatibility with the UI. I was initially planning on a compatiblity layer, but I think it would be better to update the UI for the new types. Let me know if I can help with that in any way.

yqf3139 · 2017-07-26T16:11:09Z

@soamvasani Thanks. I made some experiments with TPR in fission-benchmark. I will work on the new types.

soamvasani · 2017-07-26T16:15:15Z

Issues #103, #129, #184, #246.

erwinvaneyk · 2017-07-26T22:59:57Z

types.go

+	// "sha256" is the only currently supported one. Sum is hex
+	// encoded.
+	Checksum struct {
+		Type string `json:"type"`


You might want to alias this to ChecksumType, similar to PackageType

erwinvaneyk · 2017-07-26T23:04:27Z

types.go

+		Literal []byte `json:"literal"`
+
+		// URL references a package.
+		URL string `json:"url"`


I have not read through all the changes yet, but why do you need the distinction here between literal and URL? That might be something to extract from the actual Package struct to something like this:

PackageLocation (Url/Filesystem) ---fetch--> Package

That would make the package agnostic of what way it was retrieved.

erwinvaneyk · 2017-07-26T23:30:44Z

types.go

@@ -92,6 +175,52 @@ type (
 	}

 	errorCode int
+
+	//
+	// Fission-Environment interface. The following types are not


Is this just about FunctionLoadRequest? For clarity it might be an idea to split these internal type definitions to a separate file/package.

Yes it is; I'll split it out.

erwinvaneyk · 2017-07-26T23:31:47Z

types.go

-		Metadata             `json:"metadata"`
-		RunContainerImageUrl string `json:"runContainerImageUrl"`
+	// FunctionSpec describes the contents of the function.
+	FunctionSpec struct {


Is this the definition of a function in general or the definition of a (deployed) function instance?

It's the definition of a fission function. Not sure I get the question, what do you mean by "instance"?

Ah, I thought there was also a plan to introduce a declarative function definition (something like the serverless.yml file in the serverless framework or npm's package.json). I thought this was it. :P So how do users have to specify package metadata (like the package entrypoint and version) now?

This is part of the plan for declarative definitions. You can have a yaml file that's the serialized version of these objects, just like any other kubernetes yaml files. (Client tooling to work with those files will be improved in future changes.)

All types now have a spec, following the pattern of K8s objects. Functions now have source and deployment packages. A Package can be specified by literal, or by URL. Environments now have a builder and runtime component. All triggers use a new FunctionReference to specify the function. This for now only uses a function name, but in the future can be extended to be more flexible. A new FunctionLoadRequest type is added for specialization requests to the environment runtime.

Implements TPR types using the spec types in fission/types.go. Adds code for adding creating TPR types, and convenient types for crud operations on each of our resource types. Adds code for connecting to K8s API and configuring a REST client with fission types set up.

This apiserver is now simply a stateless api layer on top of the TPR types. At the moment it doesn't do anything that couldn't be done by simply talking to the TPR types. In the future we can have better validation and potentially some higher level APIs (like versioning for example) in here.

As far as possible we keep the CLI flags the same. We'll have to add flags for source/deploy packages and builder/runtime environments. That will come in the next change.

Also adds a poolmgr_test.

Also adds a function reference resolver, which separates out the job of resolving a FunctionReference to a function.

Remove controllerUrl flag, since we don't need it any more.

Also update the poolmgr commandline, and use an env var for the fetcher image URL.

soamvasani force-pushed the v2-types-and-tpr branch from c6c5831 to 801de25 Compare July 26, 2017 21:52

erwinvaneyk reviewed Jul 26, 2017

View reviewed changes

erwinvaneyk mentioned this pull request Aug 4, 2017

source_to_image integration #268

Closed

soamvasani added 13 commits August 5, 2017 00:46

Split controller client into files and update for v2 types.

9435297

Update CLI for v2 types.

dcd6ad4

As far as possible we keep the CLI flags the same. We'll have to add flags for source/deploy packages and builder/runtime environments. That will come in the next change.

Update poolmgr and fetcher for v2 types.

c0cd043

Also adds a poolmgr_test.

Update router for new types.

905df72

Also adds a function reference resolver, which separates out the job of resolving a FunctionReference to a function.

Update kubewatcher and timer for v2 types.

ea52c1a

Update Message Queue trigger type for v2 types.

64d279c

Minor odds and ends.

e5f5a81

Fission bundle CLI updates

fc475b8

Remove controllerUrl flag, since we don't need it any more.

Remove etcd deployment (replaced by storing state in TPR)

9ecfbfe

Also update the poolmgr commandline, and use an env var for the fetcher image URL.

Explicit ChecksumType and consts

e55e1b4

soamvasani force-pushed the v2-types-and-tpr branch from 7f896d2 to e55e1b4 Compare August 5, 2017 07:58

Clarify separation of environment interface types

5142e29

soamvasani merged commit e238776 into master Aug 5, 2017

darkgerm deleted the v2-types-and-tpr branch May 25, 2019 06:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

V2 types and TPR #266

V2 types and TPR #266

soamvasani commented Jul 26, 2017 •

edited

soamvasani commented Jul 26, 2017

yqf3139 commented Jul 26, 2017

soamvasani commented Jul 26, 2017

erwinvaneyk Jul 26, 2017

soamvasani Jul 26, 2017

erwinvaneyk Jul 26, 2017

erwinvaneyk Jul 26, 2017

soamvasani Jul 26, 2017

erwinvaneyk Jul 26, 2017 •

edited

soamvasani Jul 26, 2017

erwinvaneyk Jul 26, 2017

soamvasani Jul 27, 2017

V2 types and TPR #266

V2 types and TPR #266

Conversation

soamvasani commented Jul 26, 2017 • edited

soamvasani commented Jul 26, 2017

yqf3139 commented Jul 26, 2017

soamvasani commented Jul 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erwinvaneyk Jul 26, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

soamvasani commented Jul 26, 2017 •

edited

erwinvaneyk Jul 26, 2017 •

edited