Evaluation

An Evaluation of Express as a Platform as a Service

Introduction

One of the major focuses in software development has been a shift towards rapid application development. This corresponds to the emergence and growth in the popularity of platforms as a service (PaaS) and backends as a service (BaaS). Platforms as a service allow for applications to be deployed without having to worry about any of the underlying architecture. This often comes in the form of one-click deployment and automatic server provisioning. These services abstract over the architecture that web applications and APIs run on to allow developers to focus more directly on building their applications. Backends as a service provide an additional layer of abstraction by automatically providing additional services like database and user account management.

As a result, these services allow developers to devote even more time towards developing the unique aspects of their applications rather than re-inventing the wheel. The rapid growth in the popularity of PaaS and BaaS solutions suggest that there is significant demand amongst the development community. However, existing solutions suffer from problems like high cost, vendor lock-in, and lack of supporting services causing developers to spend significant amounts of rebuilding components like user and account management. Express is a new PaaS that seeks to solve these problems by providing an open source solution that can be deployed to any cloud or bare metal hosting provider, abstracts over deployment and scalability, and provides built-in support for common services like user and account management.

Background

One major category of PaaS and BaaS solutions consists of commercial services like Google App Engine, Heroku, Firebase, and Amazon Elastic Beanstalk. These services all offer conveniences like automatic provisioning, horizontal scalability, and database management. The problems arise when examining their cost structure. Many of these services offer free tiers making them extremely economical for early development and launch (Google, 2018). However, once the need to scale arises costs rise rapidly. One Heroku user reported that switching to managing their own servers and infrastructure reduced their monthly costs by four times (Bigdeli, 2017). The second problem arises with vendor lock-in. While some users may be able to invest the time to transition their applications away from running on a SaaS or BaaS, many cannot due to vendor lock-in. Many applications require significant redesign due to being built around the proprietary user management, infrastructure, and database solutions offered by these services. As a result, with the commercial solutions, it can be both too expensive to continue using them and too expensive to leave due to the redesign.

The second major category of PaaS and BaaS solutions consists of open source projects. Many of the projects are in still early in development; however, a few have been gaining popularity. Examples include Dokku, Flynn, and Deis. The primary problem with these solutions is that they focus on abstracting over the infrastructure without providing support for additional services. They allow for databases and applications to be deployed using single commands but lack solutions for extremely common tasks like authentication and user management (Viray, 2017). As a result, developers that use them can save money and eliminate vendor lock-in but still end up spending a substantial amount of time re-inventing the wheel due to having to build services for user management. A couple of solutions exist that do offer these additional services like Laravel Spark; however, they do so at the cost of no longer being language agnostic (Laravel LLC). As a result, there is a significant gap in the market for open source PaaS and BaaS solutions.

Related Work

In this section, existing PaaS and Baas solutions will be analyzed to see their benefits and shortcomings. This focuses on both commercial and open source solutions.

Heroku

Heroku is one of the most popular language agnostic PaaS solutions. It abstracts over the infrastructure and deployment by providing a command line interface for single command deployment. It also supports automatic scaling and database management. The primary issue is in the form of cost with the hosting of an application and database for professionals starting at $25 and $50 per month respectively (Salesforce.com). These costs can quickly become prohibitive. Even with these high costs, Heroku lacks solutions for common services like user account management instead requiring developers to design their own solution.

Firebase

Firebase is a popular BaaS solution that offers a higher level of abstraction that Heroku. It saves developers time by offering services like user authentication and push notifications in addition to abstracting over the infrastructure and deployment of applications. However, these conveniences come at the cost of vendor lock-in and being subject to sudden price changes. Firebase offers proprietary closed source database, hosting and user management solutions and have complete control over pricing. One user reported sudden pricing changes causing their expenses to increase 7000% overnight (HomeAutomation, 2017) which has the potential to put a company out of business if they cannot afford to redesign their applications to switch away from proprietary services.

Flynn

Flynn is an emerging open source PaaS that abstracts over the underlying architecture by allowing single command deployment of applications and databases. This allows developers to keep costs significantly lower than Heroku and Firebase while also avoiding vendor lock-in by choosing their preferred host. However, it does not provide any solutions for common services like user account management meaning developers have to invest time into developing user account models for their database, endpoints for sign in and registration, and security for their authorized endpoints.

Laravel Spark

Laravel Spark differs from the other solutions because it focuses more on offering services like account management and payment. However, it lacks many of the important features offered by other solutions like being language agnostic and providing automatic horizontal scalability. As a result, developers can save time and money for smaller scale projects but may run into problems when looking to incorporate libraries or tools that do not support PHP or scale up their application across multiple machines to handle a high number of users.

Solution Description

The Express platform offers an open source solution to the shortcomings of the related PaaS and BaaS solutions by abstracting over the underlying infrastructure with single command application deployment and autoscaling while also providing built-in services like account management. The Express platform is built as a layer of abstraction on top of the open source Kubernetes project. As a result, its features consist of building on top of the deployments, stateful sets, ingress controllers, and autoscalers offered by Kubernetes. Using Kubernetes as a base allows Express to be language agnostic via containerization and deployed to any cloud or bare metal hosting provider. Additionally, this means that developers using Express can access the underlying infrastructure offered by Kubernetes to extend the platform if they find their needs extend beyond what is offered.

The Express platform solves these problems using three main components: a load balancer and gateway router, an authorization and identity service, and the Express command line interface.

Express uses NGINX for load balancing and routing with one instance running per machine in the cluster. The load balancer and gateway router is vital for providing both automatic scalability and user management. It is responsible for load balancing requests between multiple instances of the authorization and identity service as well as user deployed applications. This allows applications deployed by developers using the command line interface to seamlessly scale across multiple physical machines in the cluster without requiring any management of the underlying platform or architecture. The NGINX configuration is controlled by a set of Kubernetes ingress controllers. These Ingress controllers are dynamically created by the Express controller when using the deploy command to deploy a web application or API.

The gateway router also plays a vital role in providing seamless user management and security without developers having to worry about protecting their deployed applications. When developers deploy their applications, they can provide public endpoints configurations to specify which endpoints should be exposed and which scopes they require. This allows the gateway router to intercept the incoming request, check which scopes they require, and route them to the authorization and identity service to verify their access tokens before forwarding them to the developer deployed applications. As a result, developers can trust all traffic that reaches their deployed applications due to it having been filtered and authorized by the gateway router.

To allow the gateway router to authorize requests, scopes are built into the NGINX routing configuration. A scopes HTTP header is added to all requests that require authorization by gateway router. The scopes are determined based on being statically defined on a path by path basis in the NGINX configuration. This ensures that the gateway does not have to do any additional looking up scopes for requests that require authorization. The headers of these requests containing the scopes and access token are then proxy passed to the authorization and identity service. If the response is accepted the user account information is returned for it to be injected into the HTTP header of the original request which is then forwarded to the internal service.

The authorization and identity service provides the built-in endpoints for account login, registration, and token verification. This means that developers do not have to do any work in order to get access to user account management thus allowing them to focus purely on developing their applications. Having this built into Express also means that it can be explicitly designed to scale horizontally. It is built on top of a Cassandra database allowing for both data redundancy to avoid loss if a machine in the cluster goes down and to ensure the platform scales horizontally.

Unlike the authorization and identity service, the Cassandra database is not attached to a Kubernetes autoscaler. During trials, the Cassandra database was unable to handle increases in load due to having to stream data to new nodes before scaling. This delay in scaling meant that the Express platform was unable to handle rapid increases in load. Because of this, the database is designed to run a single node per physical machine in the cluster. This design slightly increases resource usage across the platform but allows it to handle rapid increases in load without losing stability.

The command line interface provides the necessary abstraction required for developers to deploy, update, and remove their applications with singular commands. It ensures that the load balancer and gateway router get properly configured to both forward and authorize requests to developer deployed applications. It is also important for creating the necessary deployments and autoscalers to ensure that the number of user deployed application instances can be increased and decreased as the load on the platform in terms of the number of requests per second changes. The command line interface relies on the Kubernetes API to interact with the cluster. Through this API it coordinates the creation of deployments to run deployed applications and ingress controllers to reconfigure the gateway router.

As a result, the load balancer and gateway router, authorization and identity service, and command line interface work together to ensure that Express offers a solution that solves the problems currently faced by existing PaaS and BaaS solutions. All three are required to ensure that horizontal scalability, user management, and deployment are all offered at a sufficient level of abstraction to allow developers to focus on rapid development of the elements that make their applications unique.

Experimental Setup

The goal of the experiment was to evaluate the Express platform in a production environment with realistic traffic. This experiment focused on evaluating the platform's ability to scale horizontally rather than the functionality of the Express controller, and authorization and identity service. The Express controller and authorization and identity were tested before this experiment using unit tests to ensure that they were functioning properly.

Express Platform

The Express platform was deployed to a Kubernetes cluster with one to eight machines each with their own dedicated resources. Having their own dedicated resources allows the Express services to be tested in a production environment. Five trials were run at each of the eight cluster sizes.

Hardware

1. Each machine in the cluster was a Compute Engine instance with two vCPU cores and four GB of memory in the us-central1-a zone.

Software

The machines running the Express platform met the following software requirements.

1. Kubernetes 1.8.5. A Kubernetes cluster running version 1.8.5 deployed using Google Kubernetes Engine with master nodes isolated from worker nodes.
2. Helm v2.8.0. The Helm server was initialized on the Kubernetes cluster using the Helm version v2.8.0.
3. Express v1.0.0-alpha.1. The initial Express platform's alpha release was installed on the cluster.
4. Test API v1.0. The test API developed to evaluate this project was deployed to the Express platform. The public v1.0 published to Docker Hub was deployed to the Express platform.

Test Driver

The test driver was run on a separate machine that communicates with the Express services over HTTP. Using this approach ensured that the test driver did not have any impact on the performance of the Express platform running on the cluster.

Hardware

1. The test driver was run on a Google Compute Engine instance with 16 vCPU cores and 16 GB of memory in the northamerica-northeast1-a zone.

Software

The test driver was running the following software:

1. Bash 3.2.57(1)-release.
2. Python 3.0.
3. Locust v0.8.1. The public Locust v0.8.1 fork developed specifically for these experiments with support for additional data logging. This was run in distributed mode with 1 master and 16 slaves.
4. Locustfile. The Locustfile developed to model multiple users interacting with the test API.

Test Application

To evaluate the Express environment, a test API needed to be developed to deploy within it. A create-read-update-delete (CRUD) API to search for campgrounds within Canadian National Park was created for this purpose.

The test API has the following endpoints.

1. /campgrounds/v1/add Allows an account with admin scope to add a new campground including the that can be included in scan requests.
2. /campgrounds/v1/list Provides a list of all campgrounds that can be scanned for.
3. /searches/v1/add Allows an account with user scope to add a new scan request to search for an available campsite from a list of provided campgrounds for a given date range.
4. /searches/v1/list Allows an account with user scope to list their scan requests.

User Model

To use this approach to test the Express services, a user model was created to simulate the HTTP requests that a user might send. This user was designed to interact with the test API and authorization / identity service in a pattern simulating real use. To achieve this, the following model was developed.

• /searches/v1/list
  • Weight: 30
• /campgrounds/v1/list
  • Weight: 30
• /searches/v1/add
  • Weight: 15
• /oauth2/login
  • Weight: 2
• /oauth2/refresh
  • Weight: 2
• /oauth2/register
  • Weight: 1

With this model, the simulated users register accounts, login, refresh their access token, list the available campgrounds, list their requested searches, and add new searches. These requests are sent randomly; however, requests with higher weights are sent more frequently.

Common requests like listing the searches and campgrounds are weighted much higher to simulate a higher frequency. Logging in and refreshing a new token have much lower weights to reflect that they are performed less frequently. Registration occurs the least frequently; however, the simulated users must periodically re-register to avoid continually growing the list of searches which would skew latency due to the response size increasing as the test duration increases.

Procedure

The experiment was initiated from the test driver once all the hardware and software requirements had been met on the test driver and Express platform cluster. The initialization and experiment steps were performed 40 times with the cluster size starting with at one machine and being increased by one machine after performing five trials at the respective cluster size. This resulted in five trials per cluster size.

Initialization

The cluster was put in a consistent state when starting each set of experiments. This was accomplished by performing the following tasks. Initialization involves working directly with the underlying Kubernetes cluster to ensure that Express platform is in the same starting state for all trials without having to explicitly re-install Express each time.

1. Removing the previously deployed application using express-controller teardown testapi.
2. Removing the Express environment from the cluster using helm delete express-test.
3. Removing all persistent volume claims using kubectl delete pvc express-test-database-0.
4. Removing all persistent volumes using kubectl delete pv express-test-database-0.
5. Waiting until No resources found. is shown when using kubectl get pods.

Experiment

With the cluster in its initialization state, the experiment procedure was executed. This was performed on the test driver by completing the following steps downloading the Express alpha release.

1. Navigating to the /integration-testing directory of the Express v1.0.0-alpha.1 download in bash.
2. Executing the command locust --host=https://<cluster-ip> substituting <cluster-ip> for the IP address or domain name of the Kubernetes cluster being used.
3. Opening a web browser and navigate to http://localhost:8089.
4. Entering 20,000 as the number of users to simulate and 1 as the hatch rate and clicking Start Swarming to start the user simulation. 20,000 is an arbitrary number of users. Any number can be picked as long as the maximum number of requests per second exceeds what kind be handled by the Express platform when running on the maximum number of machine instances.
5. Stopping the test driver when either of the cutoff criteria were met:
   • The average sliding-window response time exceeded 500 ms.
   • The proportion of failed requests exceeded 2%.
6. After stopping the test driver, the statistics were recorded from the Locust download's tab in the web browser by downloading the following files.
   • request statistics CSV
   • response time distribution CSV
   • exceptions CSV

Results

A simple linear regression was calculated to predict the platform's ability to handle requests based on the number of machines in the cluster. A significant regression equation was found with an R-squared of 0.98144. The model predicted the number of requests is equal to 62.171 + 21.176 (number of machines) where the number of machines is a discrete measure of the number of physical machines in the cluster. The number of requests increased 62.171 for each additional machine in the cluster.

These results show that the Express platform was able to scale almost perfectly as additional machines were added to the cluster without any diminishing returns. The y-intercept of 21.176 is hypothesized to be a result of being able to handle a greater number of requests running on a cluster of size one due to minimizing latency and network overhead from guaranteeing that all stages of the request are handled on the same machine.

Some stability problems were experienced at high resource utilizations as shown in the above graph of one of the 40 trials. All trials showed similar stability degradations at high resource utilizations. At approximately 45 seconds in this trial, the cluster reached 90% CPU utilization. At this point, a consistent fluctuation in requests per second appears with significant dips at even intervals. Due to having limited resources, reduced stability was expected at higher resource utilizations but not to this extent. Logs suggest that the performance dips coincide with garbage collection in the Cassandra database used by the authorization and identity service for persisting accounts and access tokens. These stability problems are not a significant problem due to only occurring at high resource utilizations; however, they are addressed in the Express roadmap.

Discussion

The Express platform has been released as a minimum viable PaaS built on top of Kubernetes. By being built on top of Kubernetes, users of the platform can deploy it on any hosting provider whether it be a bare metal or cloud provider. Additionally, using Docker to containerize applications ensures that the platform is language agnostic to allow developers to develop web applications or APIs using their preferred languages and frameworks. These two features ensure that the Express platform can be used at a much lower cost than commercial platforms due to not having to commit to proprietary services or technology.

Similar to existing open source PaaS solutions like Flynn and Dokku, the Express platform supports the full application lifecycle using a simple command line interface. This allows developers to rapidly deploy, update, and teardown their applications and APIs within the platform using singular commands. Existing open source PaaS solutions like Flynn have additional support for services like database management to easily deploy databases. In its current state, Express does not have explicit support for database management. Developers are responsible for configuring their own database containers and using the existing application lifecycle command line support. However, database management is on the Express roadmap.

The real benefit of Express over existing open source PaaS solutions comes from the support for additional services specifically focusing on identity and authorization management. Almost all web applications and APIs involve user accounts and protected endpoints to restrict resources based on being signed in. The argument for existing open source PaaS solutions is based upon using one of the many existing libraries for identity management. The problem with this solution is that it requires developers to make difficult decisions based on determining which identity and authorization library to use, how to integrate it into their web applications, APIs, and database schemas, and how to handle identity and authorization across multiple services running on a single platform. Even with existing libraries, these problems require a significant amount of time from developers that could be better spent focusing on what makes their applications and APIs unique.

Additionally, one of the most significant benefits of using Express is that integrating identity and authorization management into the platform reduces the burden of security traditionally placed directly on developers. By integrating authorization and identity into the platform's gateway router, unauthorized requests never have an opportunity to reach the internal applications and APIs deployed within. As a result, developers do not have to implement any security in their programs. They can instead trust incoming requests and use the user information that has been injected by the gateway router after authorizing incoming requests based on their access tokens. As a result, the burden on developers is significantly reduced by using the Express platform.

Future Work

Based on the Express platform's alpha release being a minimum viable PaaS solution, there is a significant amount of future work on the roadmap. These features are targeted for the platform's beta release.

Database Management

The first major feature on the roadmap is database management. Many of the open source PaaS solutions are based on support for many different databases. With Express, database management will be based on sharing one distributed database between all applications. This will be accomplished by provisioning password protected key spaces in the database on a per-deployment basis. The credentials will then be injected into the deployments as environment variables to allow for connection without any configuration by developers. Using multiple independently protected key spaces will ensure that the data stored by different applications and APIs deployed by developers using Express will be isolated and protected.

Performance Monitoring

The next feature on the roadmap is based on providing better guarantees of performance. While the evaluation of Express involved stress tests to ensure that it offers horizontal scalability without diminishing returns, the performance of the platform could be undermined by rogue applications and APIs or bad database queries. To help developers ensure that their applications and APIs do not degrade the performance of the platform, services to monitor for poorly performing database queries and applications and APIs that are using disproportionate amounts of the platform's resources. These services will help developers using the platform monitor, diagnose, and resolve performance problems.

Account Management Page Customization

Improvements for the authorization and identity service are also on the roadmap. In the alpha release of the Express platform, authorization and identity focuses on providing registration and login using publicly exposed API endpoints. Login and registration pages exist for web applications but, in their current state, only return access tokens. Improved web application support will be added in the future to allow users of the Express platform to specify a redirect uniform resource locator (URL) attached to their client identifier. Additionally, they will be given the option to specify a static page URL from which the login and registration pages will be rendered to allow for customization of the login and registration pages.

Let's Encrypt Support

In its current state, Express requires developers to manually configure HTTPS for their deployed web applications and APIs. In the future, support for Let's Encrypt to automatically create HTTPS certificates and renew them occasionally to prevent expiration. This will ensure that all traffic to and from the Express platform is secure without requiring an effort by the developers using the platform.

Stability

Stability improvements will be made to try and increase cluster stability at high resource utilizations. This will be done by benchmarking the performance using alternative garbage collection methods in the Cassandra cluster. The current configuration relies on stop the world garbage collection which may be amplifying the effect. Alternative databases that do not rely on garbage collection may also be explored. The primary candidate for this is ScyllaDB due to the ability to directly swap it out for the Cassandra database without losing scalability or changing the interface.

Conclusions

With its alpha release, the Express platform has taken the first steps towards filling the gap in the market left by existing commercial and open source PaaS solutions. It offers rapid deployment of web applications and APIs to ensure that developers spend the minimum amount of time managing deployment instead allowing it to be devoted towards development. The experiment performed for this evaluation has shown that the Express platforms ability to scale horizontally is on par with the offerings of other PaaS solutions. The Express platform provides additional benefits over existing PaaS solutions by directly integrating authorization and identity management which removes the burden of security from the developer. This allows developers using the Express platform to save a considerable amount of time and prevents them from making costly security mistakes. For this reason, the Express platform allows developers to get from development to release much faster than with existing open source PaaS solutions without the high costs of commercial solutions. Certain key features like database management are still missing from Express meaning it may be too early for some developers to gain its full benefit. However, the features on the roadmap put the Express platform in a position to significantly outperform existing PaaS solutions in their ability to reduce development time and developer cognitive burden.

References

Bigdeli, K. (2017, December 10). How I built a replacement for Heroku and cut my platform costs by 4X. Retrieved April 06, 2018, from https://medium.freecodecamp.org/how-i-cut-my-heroku-cost-by-400-5b9d0220ce13

Google. (2018, January 24). Always Free Usage Limits. Retrieved April 04, 2018, from https://cloud.google.com/free/docs/always-free-usage-limits

HomeAutomation. (2017, May 17). Firebase Costs Increased by 7,000%! Retrieved April 06, 2018, from https://startupsventurecapital.com/firebase-costs-increased-by-7-000-81dc0a27271d

Laravel LLC. (n.d.). Retrieved April 04, 2018, from https://spark.laravel.com/

Salesforce.com. (n.d.). Simple, flexible pricing. Retrieved April 04, 2018, from https://www.heroku.com/pricing

Viray, J. C. (2017, February 08). PAAS comparison - Dokku vs Flynn vs Deis vs Kubernetes vs Docker Swarm (2017). Retrieved April 04, 2018, from http://www.jancarloviray.com/blog/paas-comparison-2017-dokku-flynn-deis-kubernetes-docker-swarm/