Repository navigation
Framework
Remark: The following description relates to the project prototype developed at Saarland University. It does not describe a fully functional and implemented project.
The Analysis Bots Platform for GitHub consists of multiple parts: The platform (webapp), the workers and the bots. These parts directly relate to a multi stage process: A task/job is scheduled for execution (platform) and thus passed to an executor (worker) which then executes the task (i.e. runs a program to complete the task or in this case a bot).
This creates a separation of concerns and highly specialized domains. As a result the aforementioned stages are well defined and less error prone. E.g. only a limited number of actions must be performed by a worker in order to serve its purpose: For the given task fetch the related project from GitHub and the related bot from DockerHub before executing the bot on the project and reporting the results back.
The platform itself consists of several parts:
The Go implementation of the platform primarily serves as an HTML and Javascript server. Using a RESTful route system the user can navigate the different pages and trigger actions. A route request triggers an HTML template to be rendered (fetch and fill in data) which then is sent back to the users browser.
This project uses the Gorilla web toolkit for the routing system and for the handling of user sessions by means of cookies. The used cookies are encrypted and every attempt to change one invalidates it.
Basically, the API is a command-line interface for the platform. Most platform functions are accessible through it. On the user page authentication tokens can be registered that then can be used to identify and authorize the API accesses. Additionally, API accesses are recorded and limited to overcome the potential for abuse.
Similar to the webfrontend the API is RESTful and as it is considered to be used on the command-line or by third-party applications it relies on JSON-objects for data transfer. Please, review the source code for details.
Firstly, the backend is based on these third-party services:
- PostgreSQL Database for persitent data storage.
- GitHub API for the acquisition of user and project data as well as the creation of pull requests and webhooks.
- DockerHub API for sanity checks during the registration of new bots.
For the data layout and the exact API calls, please review the source code.
Furthermore, an RPC server for request of the workers and a scheduler for task executions belong to the platform's backend. The RPC server simply translates and forwards the RPC calls to the rest of the platform and maintains a list of available workers.
Lastly, the task scheduler follows a very simplistic design, too. Each new task creates a Go-routine that sleeps until a point in time where the tasks shall be executed. Then the Go-routine triggers everything to execute the task (i.e. wake a suspended worker and pass it the task description). If the task is of type "Scheduled" (i.e. it is repeated based on time events) the Go-routine sleeps again until the next event.
As described above, workers serve one purpose: Execute a given task and report the execution results back. The worker prototype is implemented in a separate project which can be found here. Following a brief summary of the worker functionality:
The worker uses Go's net/rpc package for the communication with the platform. Before starting a worker it needs to be configured with the appropriate server data (host name and port). On startup the worker tries to contact the server in order to report itself as an available worker.
On the first startup something different happens: The user is prompted to enter a variety of data including a registration token (user unique, generated on user registration), a name and whether this worker should be shared with other users. In case the user enters the data correctly the worker is able to register itself as such at the platform from which it receives its own token that uniquely identifies the worker among all other workers. The received token is stored on the hard drive to enable the worker to identify itself on the next startup. For now only admins are allowed to create shared workers, hence the registration of a shared worker by a non-admin user will fail. After a successful registration the worker will show up on the users page and can be unregistered there.
Each worker will register itself on startup and unregister on shutdown which is to say that it is available and unavailable, respectively. Available workers will perform an infinite loop asking for new tasks. If there is no new task, the call is suspended until there is one. Otherwise a task description is returned by the platform that contains all information necessary to execute the task.
A task's execution is straightforward: The worker pulls the newest Docker image from DockerHub. The name of the image is part of the task description as well as the URL of the GitHub project on which the bot shall be executed. Then the project itself is cloned into the workers project cache directory. If the bot is able to create Git patches, a unique directory is created in the worker's cache that can be used by the bot to store the patch file.
Next, the bot is executed on the project using Docker. The local project cache directory is mounted read-only into the Docker container with removed host information of the origin remote. This way no bot has access to the user's GitHub token. So that the bot is aware were the project directory is mounted, the first command-line argument of the bot is the path to the directory inside the container.
Finally, the project results are submitted to the platform. This includes the outputs from stdout and stderr, and if applicable the patch file.
Bots can be arbitrary Docker images as long as they comply with the following rules:
- The bot outputs its results on the console (
stdoutandstderrare differentiated by the platform). - The bot expects a single command-line argument that indicates the directory containing the project on which it should work.
These rules apply to all bots that can be used on the platform. Additionally, the platform supports bots that create Git patch files (containing fixes etc.). These bots need to comply with another rule:
- The second command-line argument indicates the path and file name where to store the Git patch file data.
A collection of sample bots can be found here.