Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Appwrite 'Cloud Functions' #307

Closed
eldadfux opened this issue Apr 1, 2020 · 29 comments
Closed

Appwrite 'Cloud Functions' #307

eldadfux opened this issue Apr 1, 2020 · 29 comments
Assignees
Labels
enhancement New feature or request

Comments

@eldadfux
Copy link
Member

eldadfux commented Apr 1, 2020

Adding cloud functions is one of the most requested Appwrite features to date. Before we finalize the feature spec, we would love to get as many community feedback and suggestions as possible.

Goal

The goal of this feature is to create a sandbox where user-generated code can execute both on-demand and on existing system event (user.create, document.update, file.create, etc...). Each sandbox will be provided with both event-specific data and user provider env vars. This feature will help users trying to avoid the overhead of setting up a full backend server alongside Appwrite for specific custom tasks.

Cross Platform & Language Agnostic

On of Appwrite main goals is to stay cross-platform and framework agnostic. We treat all tech stacks equally, so this feature should allow us to run code from multiple languages. We would probably support Node.js, PHP, Ruby, and Python first, but potentially any language can be supported.

Security & Isolation

We are going to use Docker and Appwrite built-in queuing engine to execute the tasks in a completely and isolated environment.

Performance & Scalability

To make sure performance is optimized, we will review the possibility to pre-download the core docker images so run time should only be depended on the code running.

All tasks will run in async way using our job workers. This will also allow us to scale very easily by adding new workers as demand grow.

Abuse Protection

We should possibly also create some sort of a timeout to help protect smaller Appwrite instances running on a single server from being abused.

Packaging

This is one of the bigger question marks. Since we need to support multiple languages and make sure the final package can be easily executed using a base Docker image, we need to allow developers to package and upload their code easily.

Some open questions:

  • Should we create a CLI for uploading code packages?
  • Can we create a single CLI that will be able to deal with different coding languages? (Should we use Docker to package the CLI?)
  • Should we integrate with VCS directly? (can be very time consuming for v1 of this feature)
  • Should we support version control?
  • How do other cloud function services manage code uploading?
  • Should we allow built-in IDE in Appwrite dashboard for simple functions?
  • Should we store the code in an encrypted way? How will this influence performance and time-to-start?
@eldadfux eldadfux added discussion Gathering community feedback enhancement New feature or request help wanted Extra attention is needed labels Apr 1, 2020
@eldadfux eldadfux self-assigned this Apr 1, 2020
@TorstenDittmann
Copy link
Contributor

TorstenDittmann commented Apr 1, 2020

Then I'll add my mustard, as we say in Germany.

What bothers me about most cloud functions is the unnecessary complexity. In my experience, the requirements for most cloud functions are relatively simple.

Maybe it is because I try not to push complex functions to serverless or cloud functions.

Languages

To simplify maintainability, I would advise to stick with Node.js, PHP, Ruby, and Python only. These are probably the most common languages in that environment, except Java maybe.

Because of CLI

In my experience, these functions are not often modified. Since this entails a lot of rat-tails, I would rather recommend administration in the AppWrite Dashboard. Things like authentication, publishing and maintenance of such a CLI could be prevented. Above all, the Dashboard provides a wonderful foundation. There are also very good libraries that provide a simple code editor with highlights etc. for the web.

Unfortunately I can't say anything about performance, stability, security and scalability.

Visual Programming

Screenshot from 2020-04-01 10 56 11

I have never really seen this except in the educational field, until I came across it by chance at the provider Backendless. I smiled at it until I tried it. In 2 minutes I had a script to make backups of my user data. There are code blocks for all common programming functions like conditions, loops, timers etc. There were also blocks for all API features.
Google actually has done great work with Blockly in that area already. Blockly is a library that adds an editor to your app that represents coding concepts as interlocking blocks. It outputs syntactically correct code in the programming language of your choice. Custom blocks can be easily created for API features. This would be a great and easily maintainable alternative for the user.

Sources

@eldadfux
Copy link
Member Author

eldadfux commented Apr 2, 2020

@TorstenDittmann thanks for the great feedback! Really love how you put your focus on simplicity. I think this is one of the core values of Appwrite.

I agree that having a small IDE on the dashboard could be great for small, simple functions, but I am not sure we'll be able to avoid some kind of a deployment tool for functions that will need to rely on dependencies.

Visual programming sound interesting, but I am not sure it will be wise to tackle this concept at this stage.

@eldadfux
Copy link
Member Author

eldadfux commented Apr 2, 2020

Some visual thoughts about the design to add to the discussion

image

@TorstenDittmann
Copy link
Contributor

Indeed, that is true. I was just relating to my personal use cases.

How my perfect scenario for more advanced cloud functions would be:

  • Have a repo for my Cloud Functions
  • Integrating the CLI with CI to build/upload to my AppWrite Instance
  • Having an API endpoint for that function (also SDK method for handling those)
  • Having the ability to easily select Cloud Functions in Tasks

Something else came to my mind as well. The ability to handle permissions for whom the functions are available would be nice.

The possibility for visual programming is also something for the future and would be wise to build on an already proven solution and should not be given higher priority than essential features.

Since I have already worked with Blockly before, I am also willing to create a Proof of Concept in the future.

@wentzlau
Copy link

wentzlau commented Apr 5, 2020

I think that "cloud functions" is a great concept to work toward. I don't think that we should settle on one programming language but make an open system where it is possible to register functions in different programming languages to be called by AppWrite.

I think that my thoughts in #264 "Add api gateway for custom business logic " actually describes how to do this. In addition to the gateway functionality, it should be possible to add hooks/callbacks for appwrite events, like "new user", "db record changed".

My suggestion in #264 is very low level where a programmer should manually register functions in Appwrite. The concepts in #264 could be backed into a more userfriendly approach like this (python):

from appwrite.functions import AppwriteEvents

# setup framework, connect to app write server with api key
# this could also be configures via enviroment variables
appwrite_events= AppWriteEvents("https://appwrite.io", "apikey")

@appwrite_events.on_new_user
def new_user(new_user):
    #code to be executed when a new user is created in app write

or make a function available to a frontend app:
https://appwrite.io/my_api/my_app_write_function?param1=x & param2=y

from appwrite.functions import AppwriteFunctions

# setup framework, connect to app write server with api key and gateway route
# this could also be configures via enviroment variables
appwrite_functions= AppWriteFunctions("https://my_appwrite.mydomain.com", "apikey", "my_api")

@appwrite_function
def my_appwrite_function(param1, param2):
    #code to be executed when invoked

@appwrite_function
def another_appwrite_function():
    #code to be executed when invoked

With my approach it is possible to run appwrite functions locally on a developer machine. And upload them to an appwrite server via some kind of CLI.

The first step is to create a appwrite api to register functions and hooks.
Then to implement api gateways
From there it is possible to make specfic high level abstractions to ease creating hooks and functions.

And finally to create a appwrite cli tool for easy deploying code to appwrite docker service stack and runtime handling in appwrite to execute code in a runtime environment.

@TorstenDittmann
Copy link
Contributor

@wentzlau concept sounds great, but wouldn't this create another layer of complexity for the developer in the end?

Writing a script with the already existent Server SDK, hosting it on Appwrite and calling it with the server/client SDK or an API route seems more straight forward to me.

This could be extended with Webhooks and Tasks.

@wentzlau
Copy link

wentzlau commented Apr 5, 2020

Mostly no, it will reduce complexity. If you want to create extra functionality that is not a part of appwrite and should be called from an app/web client you need to spin up an extra service for the extra functionality, implement authentication, work with certificates and so on. In your client app you will have an extra api service to connect and authenticate too.

@wentzlau
Copy link

wentzlau commented Apr 5, 2020

@TorstenDittmann Maybe I misread your comment. I now realize that you maybe mean it is easier to handle if the code exists in a database in appwrite and executed in an runtime enviroment in appwrite.

You still have to do some bootstrapping of the code to inform appwrite where to start your code. Actually my suggestion for api sketch above would be almost identical but the code behind would be different instead of registering functions in appwrite it now informs the runtime environment where to find functions and hooks.

I have been working with aws serverless functions and online editing of functions is only useful for very small projects. It is possible to make python projects outside aws and upload them but it is very complex and has a steep learning curve.

To create a runtime environment in Appwrite to handle uploaded code is a very complex thing and I think that it is way more complex than my suggestion with a gateway api. Creating an code editing UI is also a very big task to undertake.

With my approach, the functions and hooks are more like microservices and could be hosted as docker instances. In the beginning, it is not even necessary for appwrite to have a cli to upload and handle code a microservice could be added to the Appwrite docker-compose file.

@eldadfux
Copy link
Member Author

I think having a CLI to deploy the code will probably be easier for us to code than having an online editor. On the other hand, having an online editor will definitely make things easier for a lot of developers. I'd probably want to test both directions and hope we can overcome the difficulties.

I would try to avoid adding other functionality like trying to analyze the use of code or writing more dedicated SDKs just for cloud functions. I guess the easiest and more flexible way is just to allow developers to use the Appwrite existing SDK from inside their own cloud functions. Maybe we can provide an auto-generated API Key and event information as environment variables inside the docker sandbox to help developers make decisions in their functions.

Having the same permission management as we have for files and documents could be a nice addition, and will also allow us to execute functions from the client-side. Maybe we could also allow sync or async execution?

We'd probably also want to handle some abuse control to avoid recursive function execution and avoid server burnout.

The API Gateway is a great feature, and I think Appwrite functions might allow us to gain some of the advantages that it can offer. It would be easier for us to treat them as two separate features, although one might hack his way using the cloud functions capabilities to gain proxy like behavior.

@TorstenDittmann
Copy link
Contributor

TorstenDittmann commented Apr 10, 2020

I agree on everything you said @eldadfux !

But I would prioritise a CLI over the editor for one particular reason, Deployment. Something like an editor is just really handy for developers with less backend & devops experience. Also overengineering that feature in general might be unnecessary. IMO everyone creating a big project, where complex cloud functions and workflow is required probably ends up writing his own service for their probably very specific needs anyway.

Don't know how/if this effects the language agnostic goal, but I would suggest having an Express like syntax:

//utils.js
export const getName = (req, res) => {
  const name = req.query.name
  res.status(200).send(`Hello ${name}!`)
}

Ends up as https://appwrite.io/v1/functions/utils/getName?name=Torsten for example.

Also having a Server SDK Client instance provided already for the project would lead to less headache.

The API Gateway surely is a good selling point for a more corporate audience in the future.

@javib51
Copy link

javib51 commented Apr 14, 2020

Hey! I'm new here but I just got curious by this conversation.
I like the idea of having 'cloud functions' for appwrite. I have some thoughts already:

  • I don't think that implementing a serverless engine is a good idea. It will add a tremendous overhead to the development of appwrite that I believe it's not necessary nowadays. There are multiple engines that you can integrate with. Personally, I would go for making appwrite kubernetes native and integrate it with knative. In this way, you allow an easy interface for developers but Ops can maintain something robust and well known on the industry.
  • If you decide to use knative, I believe that you can just use kn to get all the CLI serverless functionality.
  • Do not manage function at language level, do it at OS level (docker container). It will simplify maintenance and reduce development overhead. You can add important data and secrets via injected env vars and files on the container. So, developers can manage the integration and the current server SDK would be good enough without needed changes. This will also give more flexibility to developers, so they are not locked.
  • The main challenge for me is to build the container. It will require credentials to a container registry (rw permissions) and a CI system. Container registry credentials can be requested during 'cloud functions' functionality enabling flow but for the CI, you can check tekton. Tekton works fine but It's on an early stage. With tekton, you could trigger a standard pipeline that you defined previously and make everything automatic for the user.
  • I like the idea of having a CLI for it. It gives flexibility to the developer and allows CI pipelines to use it.

I hope that my thoughts were helpful :)

@eldadfux
Copy link
Member Author

eldadfux commented May 5, 2020

Just finished the first draft for the API spec:

image

The complete source code for the controller is available here:
https://github.com/appwrite/appwrite/blob/functions/app/controllers/api/functions.php

@punitdiwan
Copy link

I thought to do some research regarding availability of cloud functions existing solutions. I consider to reuse the existing solutions. I have looked into openfaas, kubeless iron fuctions etc. Almost all uses kubernetes cluster. I thought to share my observation on this.

I have dome some research on openfaas. Though it supports many languages and have API access, but the problem is that if uses docker container for each function deployment and uses docker hub as registry. Does it mean, if I have 25 functions, I have to deploy 25 containers. Our Appwrite has multiple projects. Defiantly if will grow as robust multi tenant BaaS solution. If 5 Project Owners deployed just 5 function each, it will be 25 containers hanging in our kubernates cluster.

  Then we have to wipe it off after certain period of inactivity. It will increase the complexity manifold.  What do you say members regarding this. I will look deeper in OpenFaas to know does the Openfaas stops containers after some time or not.

Regards
Punit Diwwan

@int-20h
Copy link

int-20h commented May 29, 2020

Having these 'Cloud Functions' will be great. However, whenever someone wants to do good amount of work or processing for each of the event, this can become heavy duty for AppWrite. I would really like to see leveraging Redis as pub/sub for those events. That way, I can use whatever languages/libraries I'm used to. This way, these processing are done outside of AppWrite and I can subscribe to any desired event.

@eldadfux
Copy link
Member Author

eldadfux commented May 30, 2020

@int-20h this is exactly what we plan to do. We already use Redis as a pub/sub for other internal work and we are planing to leverage this for the cloud functions as well, that way each Appwrite instance will be able to managed load and scale very easily for huge amount of background jobs.

@eldadfux
Copy link
Member Author

@int-20h we plan to run functions code inside docker images for isolation. That way no developers with you on the Appwrite platform will have access to the host machines, and you can be protected from any malicious functions.

@eldadfux
Copy link
Member Author

Obviously this will also allow you to run code in any language you wish.

@cholojuanito
Copy link

I'm sure you guys want to make a home grown solution for cloud functions but there is a very popular option called OpenFaaS, https://www.openfaas.com. Perhaps it would be possible to integrate with OpenFaaS, only if it made things easier of course.

@eldadfux
Copy link
Member Author

@cholojuanito, obviously this is something we have considered heavily. It seems that since Appwrite already have most tools set in place (redis pub/sub, usage stats tracking, heavily relied on docker, and other), the cost of making native solution will be a lot smaller compared to introducing an external solution.

Plus the fact is that most of this 3rd party solutions are heavily thighed to other systems might make Appwrite very "heavy" for smaller setups, integrating with internal events system, usage calculation and other will also have a lot of added complexity.

@eldadfux
Copy link
Member Author

Hey guys, Cloud Functions feature is progressing really well, thought I'd share some screenshots, feel free to share your feedbacks :)

image
image
image

@Hopheylalal
Copy link

Hopheylalal commented Jul 23, 2020

Functions will be use python? what about dart?

@cellerich
Copy link

To follow up on @TorstenDittmann 's saying: I will add my Swiss mustard to the conversation.

I was reading trough all the posts and agree with most of what was said. Currently I'm playing around with AppWrite for the User, Storage and Database bits and use OpenFaas on the side for functions. Everything put together in a Kubernetes cluster running in the cloud.

My wish list for functions inside AppWrite is as follows:

Languages

  • Node.js and Python (those are well used allover the communities)
  • ability to add some language specific modules/packages for my functions (I might use Pillow in Python or Selenium in Node.js)

Performance & Scalability

  • Docker (Kubernetes pods) seems the right way to go. We get isolation and can scale down to 0 if the function is used only sporadically
  • might be a Tag/Parameter for a function if we want to allow it to shutdown completely or we want to have a fast response and keep it "alive" in the background

Packaging

  • I like the concept of OpenFaas with the templates for the different languages.
  • AppWrite could provide some base containers (templates) for the different languages, maybe even with some preinstalled modules (Python-Imageprocessing = has Pillow installed, Python-ML = has Pandas, Numpy, et al installed). After adding our functions a new image could be built with a tool like Kaniko without using a Docker daemon for building.
  • However this approach needs a registry somewhere. Why not add a private AppWrite registry to the stack as well (or let users use their own registry if they have one).

Security

  • If the user brings his own docker image (and keeps his function code outside) we don't need to encrypt anything.
  • If the function code will be kept inside AppWrite (e.g. in a database) then the source should be encrypted. But this should be rather easy to do.

CLI

  • Depending on where the user should store his code this could be very helpful.
  • I fully agree with the four points mentioned by @TorstenDittmann:
    • Have a repo for my Cloud Functions
    • Integrating the CLI with CI to build/upload to my AppWrite Instance
    • Having an API endpoint for that function (also SDK method for handling those)
    • Having the ability to easily select Cloud Functions in Tasks

Integration

  • As @wentzlau pointed out, a main usability should be the easy integration of functions to events triggered by the other AppWrite functionality:
    • triggered trough user add/update/delete events
    • triggered from storage events
    • triggered from database events

Final thoughts

The maybe easiest route without building a whole CI/CD system into AppWrite would be to keep the function code outside in a separate repo. Provide a CLI to pull base images (language templates) and let the CLI orchestrate the building process. The base images might have predefined hooks for the internal "trigger events" from AppWrite. AppWrite would be responsible to instantiate the images and take care of timeouts and watchdog tasks.
Version control, source code storage and all would be outside and up to the preferences of the users.

@eldadfux eldadfux changed the title [DISCUSSION] Appwrite 'Cloud Functions' Appwrite 'Cloud Functions' Jul 30, 2020
@eldadfux
Copy link
Member Author

Thanks to all of you who posted feedbacks. We're trying to take everything into account and build a service and APIs that will give you the best ratio between flexibility, simplicity and performance.

To keep everyone in the loop, here is the up-to-date TODO list I'm working with:
image

You can also track our live progress on the swoole-and-functions branch.

Again, your feedback and questions are more than welcome both here and on our Discord server: https://appwrite.io/discord

@eldadfux
Copy link
Member Author

Following with my previous post. Here is our "getting started for cloud functions" tutorial. This is still a work in progress, but can give you a good idea about our implementation and how you could leverage the new service. Feedbacks, comments and question are welcomed.

image

@eldadfux eldadfux removed the help wanted Extra attention is needed label Aug 30, 2020
@eldadfux
Copy link
Member Author

eldadfux commented Sep 24, 2020

Status Update 24-09-2020, this are the thing left to finish before our first RC version:

  • Test timeouts
  • Add auto expiry for logs? (new daily maintenance worker?)
  • Abuse control configs
  • Tests with package managers
    • PHP
    • Python
    • Ruby
    • Node
  • Client endpoint for func execution (add new execute permissions)
  • Schedule trigger (send first event, set recurring event, cancel previous events if schedule changed)

@Acccent
Copy link

Acccent commented Oct 18, 2020

Hi all,

I hope it's ok to ask about progress here? Apologies in advance if it isn't.

I'm about to start a new project and would love to use Appwrite, but I think it will definitely need Cloud Functions in the future. For now, I can just do those in the client, which should be fine as long as we're just prototyping, but I'd like to know if there's a rough estimate of when the feature will make it into Appwrite? It's super encouraging to see this discussion and makes me even more eager to use the platform, but I also don't want to lock myself into using it if it's possible that I won't have the feature I need when I do need it, if that makes sense.

@eldadfux
Copy link
Member Author

Status Update 27-01-2021:

  • Test timeouts
  • Add auto expiry for logs? (new daily maintenance worker?)
  • Abuse control configs
  • Tests with package managers
    • PHP
    • Python
    • Ruby
    • Node
    • .NET
    • Deno
    • Dart
  • Client endpoint for func execution (add new execute permissions)
  • Schedule trigger (send first event, set recurring event, cancel previous events if schedule changed)

🎤

@eldadfux eldadfux added waiting for release Fixed or implemented and waiting for a new version to be released and removed work in progress labels Jan 27, 2021
@eldadfux eldadfux removed the discussion Gathering community feedback label Jan 27, 2021
@eldadfux
Copy link
Member Author

CLOUD FUNCTIONS HAVE BEEN RELEASED! 🥳

Full getting started tutorial:
https://appwrite.io/docs/functions

Full API specs:
https://appwrite.io/docs/client/functions

Huge thank you to everyone who helped us make this service available ❤️

@eldadfux eldadfux removed the waiting for release Fixed or implemented and waiting for a new version to be released label Jun 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

10 participants