Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Environments v2 #103

Closed
soamvasani opened this issue Feb 2, 2017 · 17 comments
Closed

Environments v2 #103

soamvasani opened this issue Feb 2, 2017 · 17 comments
Labels
Milestone

Comments

@soamvasani
Copy link
Member

@soamvasani soamvasani commented Feb 2, 2017

This is an issue that's tracking the work around redesigning the environment interface. This covers #24 but also various other issues, such as dependency resolution, better handling of compile-time errors, etc.

Current status: writing up requirements and user stories. The doc is in Documentation/wip/environments-v2.md.

@soamvasani

This comment has been minimized.

Copy link
Member Author

@soamvasani soamvasani commented Feb 2, 2017

@ktrance @dgoujard you might be interested in this issue; I will update it as we progress. If you'd like to add anything to the requirements, from a C# and a PHP point of view, this is a good place to discuss it.

@ktrance

This comment has been minimized.

Copy link
Contributor

@ktrance ktrance commented Feb 5, 2017

@soamvasani Will do.

Just looked at the AWS deployment package concept and for dotnet something like this would probably be the easiest to quickly get up and running. The developer is responsible for for packaging every component he needs which means we wouldn't have to deal with open ports to get nuget packages etc.

We could just:

  1. dynamically load the main binary either from some specification or we just scan all of them
  2. find a predefined interface/contract which we in essence have today in the dotnet env
  3. execute the function

Nice and easy. I would be interested in doing a prototype of this.

@dgoujard

This comment has been minimized.

Copy link
Member

@dgoujard dgoujard commented Feb 6, 2017

@soamvasani For PHP env ideal workflow will be :

  1. Create a zip with multiples files (php and composer.json if needed)
  2. Function on /specialize phase will install composer.json package
  3. Function will include "index.php" file on each request (and call handler if present)

This workflow did not allow to add php extension but it's more common to add package than extension (and i added many extension btw)

@soamvasani

This comment has been minimized.

Copy link
Member Author

@soamvasani soamvasani commented Feb 7, 2017

@ktrance So in this design, the user is responsible for doing the builds and creating a zip file, right? It's a good idea -- it would allow us to get started with compiled language support without solving the bigger problem of builder containers too. We still need to have a way to upload and store these zip files. I will try to spec something out for that.

@soamvasani

This comment has been minimized.

Copy link
Member Author

@soamvasani soamvasani commented Feb 7, 2017

@dgoujard I'm sorry my knowledge of PHP isn't good enough to understand this... what is the difference between a package and an extension?

We're also trying to separate out the install steps from the runtime. So basically we'd have a separate builder that would install dependencies, create some sort of package -- a zip file for example -- and then have the runtime load that when /specialize is called. And as an interim step until we have a builder, we'd require the user to package up dependencies into a zip file.

@dgoujard

This comment has been minimized.

Copy link
Member

@dgoujard dgoujard commented Feb 7, 2017

@soamvasani an extension need a compilation and loading at php startup. An package is just a library like you load in python with package.txt.

In this case the builder pod will download packages (composer.json) and the function pod will just execute code as today

@ktrance

This comment has been minimized.

Copy link
Contributor

@ktrance ktrance commented Feb 7, 2017

@soamvasani , yes that would be the general idea.

It would of course be a pain to go through if you only want to run a 5 line function but then you could still use the existing solution.

This wouldn't necessarily be the final solution but it would (likely) allow us to going very quickly and get some more experience.

@soamvasani

This comment has been minimized.

Copy link
Member Author

@soamvasani soamvasani commented Feb 7, 2017

@dgoujard thanks! I agree that leaving out extension support is fine. We can always tell users to rebuild the environment image -- that option will not go away.

In this case the builder pod will download packages (composer.json) and the function pod will just execute code as today

👍 Makes sense.

@ktrance

This comment has been minimized.

Copy link
Contributor

@ktrance ktrance commented Feb 12, 2017

@soamvasani @dgoujard

Ok, I have had the chance to read some code now and have an idea I would like to bounce of you all.

If we for a second says that we will go down the path of introducing a deployment concept which would simply mean that the code for a function isn't always a single text file but can also be a zipped archive of components.
How these components are interpreted is up to the individual environment, in the case of dotnet it could be like we discussed above a complete package we load and execute dynamically. In java a jar or something similar, in php install composer.php etc.

This could be the first step before tackling a real build component, and in all likelihood a build component would also need the ability to be fed multiple files anyway so..

One way to this would be something along these lines:

  1. add --isdeployment switch to fission/main.go, and if it is set then the --code item should be a compressed archive
  2. add IsDeployment bool to Function struct in types.go
  3. add extra json property to fetcherRequest poolmgr/gp.go function specializePod line 258,
    the property should be based on IsDeployment from Function. maybe call it unpack.
  4. add Unpack bool field to the FetchRequest struct in environments/fetcher/fetcher.go,
    consider renaming FileName to Destination as it will now work with both a file and a unzipped directory
  5. add a check in handler in environments/fetcher/fetcher.go where if FetchRequest.Unpack is true it unzips to destination,
    something along the lines of http://blog.ralch.com/tutorial/golang-working-with-zip/
  6. the enviroments then need to check for the presence of either a directory or a file during specialize post and take appropriate action

There is one caveat here and that is that the poolmgr does not, at this point in time, have any access to the Function struct.
The poolmgr only knows about the MetaData struct which does not contain information about whether or not, the Function is a Deployment or just a plain file.
Neither do I believe that the MetaData struct should contain the deployment bool as it wouldn't make any sense on for example the Environment.
The API for accessing Function data is only available in the controller's API and using that would create a weird dependency.

Thoughts?

Todo: what to do about "fission fn get" in the case of a deployment and not plain file.

@soamvasani

This comment has been minimized.

Copy link
Member Author

@soamvasani soamvasani commented Feb 15, 2017

The v2 environments doc has been updated, please take a look. Still some more detail needed in various places (feel free to send PRs for this doc, btw :) ).

I also drew some diagrams, these are the slides we had in the dev hangout from Monday; note that there are slide notes too: https://docs.google.com/presentation/d/15BcRr_9Uf-aTrfYJNjwyN_jugwafQ6IlMmJOJ9qWesA/edit?usp=sharing

cc @ktrance @dgoujard

@soamvasani

This comment has been minimized.

Copy link
Member Author

@soamvasani soamvasani commented Feb 15, 2017

Thanks for the detailed sketch @ktrance, this was helpful.

How these components are interpreted is up to the individual environment

Yup, makes total sense.

add Unpack bool field to the FetchRequest

Alternatively we could have fetcher handle the Content-Type header. If it's a zip, it should be uncompressed. Packaging format should be environment-independent (say, zip for example. All src/deployment packages can be wrapped in a zip that's handled by fetcher).

The API for accessing Function data is only available in the controller's API and using that would create a weird dependency.

I think this should be fine, poolmgr today already grabs the function object (here) when it needs to figure out which env a function belongs to, we can refactor a little bit and pass the function obj to the specialization code too.

@dgoujard

This comment has been minimized.

Copy link
Member

@dgoujard dgoujard commented Feb 15, 2017

@soamvasani How do you handle compile error feedback ? How the Buildmgr know the are errors ? And how to return this information to cli ?

Maybe the builder environement need to call Controller API to send the error message and the Controller API return the message to the cli (with websocket ?). I'm not realy fan of this idea

Or the builder environement create a special file "error.json" with error message and the Fetcher send the content of this file to the Controller API.

@soamvasani

This comment has been minimized.

Copy link
Member Author

@soamvasani soamvasani commented Feb 15, 2017

@dgoujard One way to do this is to have buildmgr capture stdout/stderr of the build command through the k8s API, and save the output into the function object by calling into the controller.

@dgoujard

This comment has been minimized.

Copy link
Member

@dgoujard dgoujard commented Feb 15, 2017

@soamvasani It seems like a good idea !

@awitherow

This comment has been minimized.

Copy link

@awitherow awitherow commented Mar 30, 2017

Accessing storage is a nice feature that is mentioned in the v2 docs. Currently using Google Cloud Platform, serving a single page app via this method is not recommended as only http is supported, not https. Any ways that fission could help alleviate this problem or is it beyond the scope/control of this project?

@Disturbing

This comment has been minimized.

Copy link

@Disturbing Disturbing commented Feb 15, 2018

Where are we at this with one?

@soamvasani

This comment has been minimized.

Copy link
Member Author

@soamvasani soamvasani commented Mar 6, 2018

This has been implemented a few releases ago.

@soamvasani soamvasani closed this Mar 6, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.