-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Appwrite 'Cloud Functions' #307
Comments
Then I'll add my mustard, as we say in Germany. What bothers me about most cloud functions is the unnecessary complexity. In my experience, the requirements for most cloud functions are relatively simple. Maybe it is because I try not to push complex functions to serverless or cloud functions. LanguagesTo simplify maintainability, I would advise to stick with Node.js, PHP, Ruby, and Python only. These are probably the most common languages in that environment, except Java maybe. Because of CLIIn my experience, these functions are not often modified. Since this entails a lot of rat-tails, I would rather recommend administration in the AppWrite Dashboard. Things like authentication, publishing and maintenance of such a CLI could be prevented. Above all, the Dashboard provides a wonderful foundation. There are also very good libraries that provide a simple code editor with highlights etc. for the web. Unfortunately I can't say anything about performance, stability, security and scalability. Visual ProgrammingI have never really seen this except in the educational field, until I came across it by chance at the provider Backendless. I smiled at it until I tried it. In 2 minutes I had a script to make backups of my user data. There are code blocks for all common programming functions like conditions, loops, timers etc. There were also blocks for all API features. Sources |
@TorstenDittmann thanks for the great feedback! Really love how you put your focus on simplicity. I think this is one of the core values of Appwrite. I agree that having a small IDE on the dashboard could be great for small, simple functions, but I am not sure we'll be able to avoid some kind of a deployment tool for functions that will need to rely on dependencies. Visual programming sound interesting, but I am not sure it will be wise to tackle this concept at this stage. |
Indeed, that is true. I was just relating to my personal use cases. How my perfect scenario for more advanced cloud functions would be:
Something else came to my mind as well. The ability to handle permissions for whom the functions are available would be nice. The possibility for visual programming is also something for the future and would be wise to build on an already proven solution and should not be given higher priority than essential features. Since I have already worked with Blockly before, I am also willing to create a Proof of Concept in the future. |
I think that "cloud functions" is a great concept to work toward. I don't think that we should settle on one programming language but make an open system where it is possible to register functions in different programming languages to be called by AppWrite. I think that my thoughts in #264 "Add api gateway for custom business logic " actually describes how to do this. In addition to the gateway functionality, it should be possible to add hooks/callbacks for appwrite events, like "new user", "db record changed". My suggestion in #264 is very low level where a programmer should manually register functions in Appwrite. The concepts in #264 could be backed into a more userfriendly approach like this (python): from appwrite.functions import AppwriteEvents
# setup framework, connect to app write server with api key
# this could also be configures via enviroment variables
appwrite_events= AppWriteEvents("https://appwrite.io", "apikey")
@appwrite_events.on_new_user
def new_user(new_user):
#code to be executed when a new user is created in app write or make a function available to a frontend app: from appwrite.functions import AppwriteFunctions
# setup framework, connect to app write server with api key and gateway route
# this could also be configures via enviroment variables
appwrite_functions= AppWriteFunctions("https://my_appwrite.mydomain.com", "apikey", "my_api")
@appwrite_function
def my_appwrite_function(param1, param2):
#code to be executed when invoked
@appwrite_function
def another_appwrite_function():
#code to be executed when invoked With my approach it is possible to run appwrite functions locally on a developer machine. And upload them to an appwrite server via some kind of CLI. The first step is to create a appwrite api to register functions and hooks. And finally to create a appwrite cli tool for easy deploying code to appwrite docker service stack and runtime handling in appwrite to execute code in a runtime environment. |
@wentzlau concept sounds great, but wouldn't this create another layer of complexity for the developer in the end? Writing a script with the already existent Server SDK, hosting it on Appwrite and calling it with the server/client SDK or an API route seems more straight forward to me. This could be extended with Webhooks and Tasks. |
Mostly no, it will reduce complexity. If you want to create extra functionality that is not a part of appwrite and should be called from an app/web client you need to spin up an extra service for the extra functionality, implement authentication, work with certificates and so on. In your client app you will have an extra api service to connect and authenticate too. |
@TorstenDittmann Maybe I misread your comment. I now realize that you maybe mean it is easier to handle if the code exists in a database in appwrite and executed in an runtime enviroment in appwrite. You still have to do some bootstrapping of the code to inform appwrite where to start your code. Actually my suggestion for api sketch above would be almost identical but the code behind would be different instead of registering functions in appwrite it now informs the runtime environment where to find functions and hooks. I have been working with aws serverless functions and online editing of functions is only useful for very small projects. It is possible to make python projects outside aws and upload them but it is very complex and has a steep learning curve. To create a runtime environment in Appwrite to handle uploaded code is a very complex thing and I think that it is way more complex than my suggestion with a gateway api. Creating an code editing UI is also a very big task to undertake. With my approach, the functions and hooks are more like microservices and could be hosted as docker instances. In the beginning, it is not even necessary for appwrite to have a cli to upload and handle code a microservice could be added to the Appwrite docker-compose file. |
I think having a CLI to deploy the code will probably be easier for us to code than having an online editor. On the other hand, having an online editor will definitely make things easier for a lot of developers. I'd probably want to test both directions and hope we can overcome the difficulties. I would try to avoid adding other functionality like trying to analyze the use of code or writing more dedicated SDKs just for cloud functions. I guess the easiest and more flexible way is just to allow developers to use the Appwrite existing SDK from inside their own cloud functions. Maybe we can provide an auto-generated API Key and event information as environment variables inside the docker sandbox to help developers make decisions in their functions. Having the same permission management as we have for files and documents could be a nice addition, and will also allow us to execute functions from the client-side. Maybe we could also allow sync or async execution? We'd probably also want to handle some abuse control to avoid recursive function execution and avoid server burnout. The API Gateway is a great feature, and I think Appwrite functions might allow us to gain some of the advantages that it can offer. It would be easier for us to treat them as two separate features, although one might hack his way using the cloud functions capabilities to gain proxy like behavior. |
I agree on everything you said @eldadfux ! But I would prioritise a CLI over the editor for one particular reason, Deployment. Something like an editor is just really handy for developers with less backend & devops experience. Also overengineering that feature in general might be unnecessary. IMO everyone creating a big project, where complex cloud functions and workflow is required probably ends up writing his own service for their probably very specific needs anyway. Don't know how/if this effects the language agnostic goal, but I would suggest having an Express like syntax: //utils.js
export const getName = (req, res) => {
const name = req.query.name
res.status(200).send(`Hello ${name}!`)
} Ends up as Also having a Server SDK Client instance provided already for the project would lead to less headache. The API Gateway surely is a good selling point for a more corporate audience in the future. |
Hey! I'm new here but I just got curious by this conversation.
I hope that my thoughts were helpful :) |
Just finished the first draft for the API spec: The complete source code for the controller is available here: |
I thought to do some research regarding availability of cloud functions existing solutions. I consider to reuse the existing solutions. I have looked into openfaas, kubeless iron fuctions etc. Almost all uses kubernetes cluster. I thought to share my observation on this. I have dome some research on openfaas. Though it supports many languages and have API access, but the problem is that if uses docker container for each function deployment and uses docker hub as registry. Does it mean, if I have 25 functions, I have to deploy 25 containers. Our Appwrite has multiple projects. Defiantly if will grow as robust multi tenant BaaS solution. If 5 Project Owners deployed just 5 function each, it will be 25 containers hanging in our kubernates cluster.
Regards |
Having these 'Cloud Functions' will be great. However, whenever someone wants to do good amount of work or processing for each of the event, this can become heavy duty for AppWrite. I would really like to see leveraging Redis as pub/sub for those events. That way, I can use whatever languages/libraries I'm used to. This way, these processing are done outside of AppWrite and I can subscribe to any desired event. |
@int-20h this is exactly what we plan to do. We already use Redis as a pub/sub for other internal work and we are planing to leverage this for the cloud functions as well, that way each Appwrite instance will be able to managed load and scale very easily for huge amount of background jobs. |
@int-20h we plan to run functions code inside docker images for isolation. That way no developers with you on the Appwrite platform will have access to the host machines, and you can be protected from any malicious functions. |
Obviously this will also allow you to run code in any language you wish. |
I'm sure you guys want to make a home grown solution for cloud functions but there is a very popular option called OpenFaaS, https://www.openfaas.com. Perhaps it would be possible to integrate with OpenFaaS, only if it made things easier of course. |
@cholojuanito, obviously this is something we have considered heavily. It seems that since Appwrite already have most tools set in place (redis pub/sub, usage stats tracking, heavily relied on docker, and other), the cost of making native solution will be a lot smaller compared to introducing an external solution. Plus the fact is that most of this 3rd party solutions are heavily thighed to other systems might make Appwrite very "heavy" for smaller setups, integrating with internal events system, usage calculation and other will also have a lot of added complexity. |
Functions will be use python? what about dart? |
I was reading trough all the posts and agree with most of what was said. Currently I'm playing around with AppWrite for the User, Storage and Database bits and use OpenFaas on the side for functions. Everything put together in a Kubernetes cluster running in the cloud. My wish list for functions inside AppWrite is as follows: Languages
Performance & Scalability
Packaging
Security
CLI
Integration
Final thoughtsThe maybe easiest route without building a whole CI/CD system into AppWrite would be to keep the function code outside in a separate repo. Provide a CLI to pull base images (language templates) and let the CLI orchestrate the building process. The base images might have predefined hooks for the internal "trigger events" from AppWrite. AppWrite would be responsible to instantiate the images and take care of timeouts and watchdog tasks. |
Thanks to all of you who posted feedbacks. We're trying to take everything into account and build a service and APIs that will give you the best ratio between flexibility, simplicity and performance. To keep everyone in the loop, here is the up-to-date TODO list I'm working with: You can also track our live progress on the Again, your feedback and questions are more than welcome both here and on our Discord server: https://appwrite.io/discord |
Status Update 24-09-2020, this are the thing left to finish before our first RC version:
|
Hi all, I hope it's ok to ask about progress here? Apologies in advance if it isn't. I'm about to start a new project and would love to use Appwrite, but I think it will definitely need Cloud Functions in the future. For now, I can just do those in the client, which should be fine as long as we're just prototyping, but I'd like to know if there's a rough estimate of when the feature will make it into Appwrite? It's super encouraging to see this discussion and makes me even more eager to use the platform, but I also don't want to lock myself into using it if it's possible that I won't have the feature I need when I do need it, if that makes sense. |
Status Update 27-01-2021:
🎤 |
CLOUD FUNCTIONS HAVE BEEN RELEASED! 🥳 Full getting started tutorial: Full API specs: Huge thank you to everyone who helped us make this service available ❤️ |
Adding cloud functions is one of the most requested Appwrite features to date. Before we finalize the feature spec, we would love to get as many community feedback and suggestions as possible.
Goal
The goal of this feature is to create a sandbox where user-generated code can execute both on-demand and on existing system event (user.create, document.update, file.create, etc...). Each sandbox will be provided with both event-specific data and user provider env vars. This feature will help users trying to avoid the overhead of setting up a full backend server alongside Appwrite for specific custom tasks.
Cross Platform & Language Agnostic
On of Appwrite main goals is to stay cross-platform and framework agnostic. We treat all tech stacks equally, so this feature should allow us to run code from multiple languages. We would probably support Node.js, PHP, Ruby, and Python first, but potentially any language can be supported.
Security & Isolation
We are going to use Docker and Appwrite built-in queuing engine to execute the tasks in a completely and isolated environment.
Performance & Scalability
To make sure performance is optimized, we will review the possibility to pre-download the core docker images so run time should only be depended on the code running.
All tasks will run in async way using our job workers. This will also allow us to scale very easily by adding new workers as demand grow.
Abuse Protection
We should possibly also create some sort of a timeout to help protect smaller Appwrite instances running on a single server from being abused.
Packaging
This is one of the bigger question marks. Since we need to support multiple languages and make sure the final package can be easily executed using a base Docker image, we need to allow developers to package and upload their code easily.
Some open questions:
The text was updated successfully, but these errors were encountered: