Feature Requests

The desirable features for the AutoLabJS software are listed below in the descending order of priority.

Installation

1. Fix the problems in the deployment tests. See the issue #317.
2. Fix the error in Vagrantfile. See the issue #327.
3. Correct the conventions in SSL config. See the issue #326.
4. Where ever possible, configure hosts using ansible_hostname Ansible variable. Also use the same technique to avoid replacing the gitlab alias of Gitlab with that of IP address.
5. Ansible playbook to check the port availability at the beginning of the installation setup.
6. Use nodemon to help restart the AutolabJS components upon reconfiguration. The suggested command is: nodemon --watch <config_dir> --watch <code_dir> main_server.js. This only requires changing the last command in docker-entrypoint.sh scripts.
   We have to be careful about hot reconfiguration. A restart will discard all the pending evaluations.
7. Right now, we have a dedicated log/ directory. Along the similar lines, can we have a dedicated data/ directory? Such a directory can have the following sub-directories.

   {{ install_dir }}/data/
       submissions/       #lab submissions
       gitlab             #gitlab data
       mysql              #mysql data
   

   Moving data to this directory makes the following directories to contain pure code and keys.

   {{ install_dir }}/
       main_server/
       load_balancer/
       execution_nodes/
   

   Once we move out the keys, we can have the pure code directories that can be easily updated.
8. Replace autolabjs-restart.sh with an Ansible script that can construct the complete container names.
9. Change the install location of node modules. This is possible using npm config. For a tutorial, see here. If we can reuse node modules across containers, that saves us a lot of space and install time.
10. Automated update script. Again, this would ideally be an Ansible package.
11. Simplified, one step installation procedure along the lines of docker installation script (see issue 131). The installation and dynamic configuration of Eclipse Che and CodaLab are worth emulating.
12. Create vagrant box and post it on hashicorp for easy demos and for use in the development of autolabcli. See this tutorial.
13. Write configparsers using Ansible's Jinja2 templates. Ansible AWX project can be used to receive user configuration inputs and then deploy AutolabJS. If we want to program even the user input collection, we can use either Tower CLI or Ansible Runner. Ansible AWX is a mix of Node.js server with Python libraries for instrumenting the Ansible backend. Even if we write a solution from scratch, it would be a poorly maintained version of the same architecture and tech stack. Where ever you can, remember to reuse the roles from Ansible Galaxy. See doc links for AWX.
    If we want a simpler graphical setup than AWX, then ara is an option. Do remember to read the security pitfalls of AWX on hackernews.
14. A script to check for installation problems. This verification script would be run immediately after installation, addition of new execution nodes or with reconfiguration of any of the AutolabJS components. This would ideally be an Ansible Tower-based package.
15. graceful failure and restart of development installation script (see issue 132).
16. Use consul for service discovery and configuration distribution. We can use Ansible's consul module to inject the configuration of AutolabJS into consul during installation. Carefully think about the starting point of configuration: it can either be inventory file or consul static config. Ansible community itself discourages putting config in inventory file, so it may be better to start with static (or generated) config of consul and pull it on demand during installation with Ansible. See the consul plugin and inventory.
17. Create a regular MySQL user with rights over the AutolabJS database. Using root database user is a HUGE HUGE security hole.
18. We are forced to use sequential execution of Ansible playbook because of access to RootCA serial file. We need to restore the property of concurrent Ansible installation.
19. We can use Alpine Linux to reduce the container image sizes. See alpine-node, alpine-java and alpine-mysql.
20. Migrate the tests/install.sh away from keys.sh to Ansible playbook-based (keys.yml) install.
21. We can use the ansible provisioning system inside vagrant to create a repeatable, easy to deploy development and installation environment. At present, vagrant.sh is doing that job.
22. Configure gitlab api rate limits to enable rapid batch processing of course enrolment.
23. Updates to Ansible installation setup.
    1. Parallelize the container building work to reduce the installation time. Ansible async and poll commands.
    2. Upgrade to GitLab API v4.
    3. Dynamic generation of keys for new execution nodes.
    4. The install script needs to consider the possibility of rerun and skip steps that have already been completed. We need to use the conditional execution feature of Ansible (import...when: clause of a task) in order to have this feature in place.
    5. Consider suggestions made in issue #260 to improve the Ansible setup
24. We can use glances to monitor the AutolabJS server. A tutorial on setup and monitoring steps for AutolabJS would be helpful.
    It might be better to put the glances installation in dependencies.yml and have an Ansible script to manage the glances.
    In case glances is not found useful, try collectd.
25. Use statsd for metric collection. See for a list of clients. Remember that prometheus and its client are way more popular and well maintained than statsd.
26. Replace MySQL with either mariadb or postgres.
27. Write an ansible playbook to create the a bundle for installing execution nodes on a different machine. The bundle must have the following.
    • Take user input: new server hostname, config of execution nodes in a truncated inventory format
    • Construct inventory from single_machine and user input
    • Create a zip bundle to be copied to new server for installation. The installation steps should be same as the normal installation steps.
28. Move environment variables of docker containers to either Dockerfile using ENV command or to docker-entrypoint.sh script. See documentation for ENV and docker-entrypoint approaches.
29. An orchestrator/agent component in a computer that runs execution nodes. The orchestrator can create / start / stop / destroy the execution nodes on demand from load balancer. This orchestrator is to be implemented using either docker-compose or ansible
30. A portable orchestrator/agent component put on a live Linux distribution with a ready to run status. The security requirements must still be satisfied. Checkout CoreOS containerized linux for this purpose.
31. Provision for environment variables where ever needed. No hard coding of filenames. Even magic constants, if any, will become environment variables.
32. One click option to put the software in maintenance mode and bring the software back to active mode. The modes to be supported are: active, maintenance, shutdown.
33. Add valid SSL certificates from letsencrypt.
34. Write configparsers. One configparser is to take in a complete config in yml file and generate separate json configs for all the AutolabJS installation and run-time; use nconf package for global config parser. Another configparser is to read json configs and help run the AutolabJS components; Use rc for the second config parser.
35. Ideal installation / update setup using vorpal.js or purely in python.

    > pip install autolabjs-admin
    > autolabjs-admin download <version>
    > autolabjs-admin configure
    > autolabjs-admin install-deps
    > autolabjs-admin install
    > autolabjs-admin uninstall
    > autolabjs-admin update
    > autolabjs-admin install enodes
    > autolabjs-admin restart -s <service_name>
    > autolabjs-admin stats -s <service_name>
    > autolabjs-admin logs -s <service_name>`      
    > autolabjs-admin install --check   #deployment checks

36. Test Ansible roles using Ansible molecule

Other Improvements

1. Create a load shedding / management module that can manage peak loads. The system works under two modes: normal and peak mode. In peak mode, the following limits will be placed on the number of evaluation requests submitted,

   • A maximum of k (with k >= 1) requests per user in one peak session.
   • A maximum outstanding requests of h (h < max. possible supported requests) at any time. All new requests beyond this point are promptly denied.
   • No acceptance of new requests while an evaluation request is pending.

   For now, a simple hack would be to provide a stock response from load balancer when the queue size reaches a maximum configured limit. The response can be a template answer which can be generated.

2. The status result from load balancer sometimes takes too long to gather. It may be better to run this periodically in the load balancer and store the result in a node-cache / local variable. When ever status request comes, we can serve the result from this cached value.

3. Timeout (3 minutes) pending evaluations in all the autolab components; timeouts avoid deadlocks. Autolabcli already has the timeout incorporated in it. The timeout at different components should be as follows: client (3 minutes), main server (2 minutes), load balancer (90 seconds) and execution nodes (60 seconds). It is better to make all the timeouts configurable.

4. User cancels an evaluation request. See issue #125

5. The code for checking the lab status (on main server) should consider four different states of a lab:
   scheduled: the lab is yet to start; hence, does not accept evaluations (status = 0, penalty = 0)
   current: the lab is in progress and any evaluation would be recorded to scoreboard (status = 1, penalty = 0)
   late: the lab has finished, but accepts late submissions with penalty (status = 2, penalty != 0)
   expired: the lab is finished and will no longer record the results of evaluations to scoreboard; the software still permits evaluations of students (status = 0, penalty = 0)
   The evaluation is to be submitted only if the lab belongs to current, late or expired. Conversely, the evaluation should not be permitted for scheduled labs.

6. After every evaluation in an execution node, the results/ directory need to be committed back to the student repository.

7. Show server time in the top-menu with a clock ticking. Use moment.js (or) luxon on the client side to fulfill this requirement. Another possibility is dayjs

8. status check on execution nodes to avoid problems during reboot of load balancer. See issue 78.

9. Use the Events module of Node.js project to decouple core-logic of an AutolabJS component from the communication module.

10. Create a generic submit() function for submitting a job to another component. Such a generic function can be reused in all the AutolabJS components. See issue 109

11. Make the repository structure semantically close with language names from drop-down menu of submit page. We can provide folder name in the brackets language name in the drop down menu.

12. Provision for explicit specification of author solutions repository. See issue #46.

13. Replace Gitlab component with Gogs. It has docker container as well. The only loss of feature is the user registration. All users need to be manually signed up. Of course, we can use LDAP authentication as well. Gogs container is 170MB in contrast with the GitLab container which is 1.3GB. There is also gitea - a fork of gog. A review.

14. The project components (main_server.js, load_balancer.js and execution_server.js) currently uses too much of low-level logic. We can create application-specific objects and use the lodash library to manage all the data structures.

15. During refactoring, consolidating all HTTPS options to one library object. See issue 118.

16. Give helpful debug log messages for two frequent mistakes -- missing file, missing directory, improper evaluation setup etc. See issue 185.

17. Preserve the last entered form settings on the submit form page. This would reduce the submission work for the student. Use cookies for this feature. See libraries - cookies.js, scott hamper.

18. Return the evaluation time along with each result. Also store this evaluation time in the database.

19. Revaluation button in admin section. Once a revaluation is requested, take the list of students from DB table, revaluate for all the commits of specified lab and add an extra column in the existing lab scores table. Once the instructor approves the suggested changes, update the lab scores column from revaluation column. See FR notes on this feature

20. Replicate ideal script structure for common admin problems. See issue 108. For us, with the presence of autolabcli, the scripts will end up being wrappers around common tasks of autolabcli.

21. NodeJS wrapper / electron-based script to initialize course, instructor, students and projects for a course. Moodle installation setup is a good example to emulate here. The script can use GitLab API to complete this step. Similar applications are: gitlab-cli 1, 2, 3. Check github for similar projects before developing this on your own.

22. Support work sample tests

23. Single commandline interface for AutolabJS administration. We can wrap container maintenance commands in this script. Ideally the autolabcli should be extended to perform this task. The script must be able to administer the manually added execution nodes as well.

24. Addition of socket.io event on client and main server to fetch the score of a user.

25. Provide log display settings for each lab. See the related notes.

26. Move the constant parts of execution setup out of lab_author solutions. The scripts that are suitable for taking out from lab_author solutions directory are: execute.sh, all language drivers, and test setup for I/O tests.

27. Keep track of AutolabJS services using ELK Stack. Add gitlab live check to status page. Gitlab has health check token and status check URLs for all the components available on https:///admin/health_check.

28. We have a clean data and execution environment separation for Gitlab container. That is, one gitlab container can be thrown away and another one can be spawned with no loss of data. Such a clean separation is missing for other containers. Once separation exists, the components would run even with old containers and updating the components would be as easy as restarting a container.

29. Use ORM module in load balancer to interact with DB. A good use of ORM module would also protect the software against SQL injection attacks. See Sequelize.js. Also migrate from MySQL to PostgreSQL. PostgreSQL supports JSON data type.

30. Log support with various log-levels using util/logger.js which is based on winston library.

31. lab name, commit hash and programming language columns in scoreboard. The scoreboard would also have title showing the lab name. If possible, we can store the user names in the lab table. These user names can be queried from the Gitlab using Gitlab API. Once the user names are available, they can be displayed on the score board. The downloaded scoreboard csv must have all the columns displayed on the website.

32. Simplify the directory structure for basic labs like "C Programming" and "Data Structures and Algorithms". If there is a lab composer for instructor, that would be useful.

33. Document source code using jsdoc.

34. Give warning message to user when a browser's back button is pressed. Inform the user of a way to go to home page. If the user still wishes to go back to a previously visited website, let the user exercise that option.

35. Refactor the execution.sh script into template approach. Provide a webpage to administrator for composing changing parts of the execution.sh. The newly created execution.sh gets committed to the lab_author/lab# repository. By default, support unit testing strategy. We may also think about migration scripts in execution nodes to a javascript/python environment.

36. Refactor load balancer into two sub-components. One is responsible for communication with other components. Another one is responsible for scheduling and dynamic provisioning of execution nodes. See related feature request.

37. Limit the gitlab repository access of execution nodes to read / commit levels to selected repositories only. That is possible to implement using gitlab API, but we are yet to implement such restrictions.

38. Run travis-ci on localhost so that the unit and functional tests can be run locally. See issue #251.

39. Change the encryption library to libsodium (NaCl). All communication between web application / load balancer / execution nodes must be using symmetric session key.

40. Subnet access control restriction for each lab. Access logs at one place for both web application / client front end and gitlab. Have a separate module to detect and mitigate DoS attacks. This module acts as a application firewall for the AutoLabJS web application. Any DoS protection module must review the best practices of application firewalls. One of the requirements here would be to translate high-level administrator directives to low-level packet-level firewall filter rules. This is possible using API gateway module such as Kong. Before using Kong API gatway, contemplate its advantages over NGINX use case.
    We can also use netmask npm package to provide the network access control. Use netmask to introduce the notion of seating arrangement and network access control to AutolabJS server.

41. Subject submission to plagiarism check using jplag. JPlag works for C, C++, C#, Java and Scheme languages.

42. Security audit of the application.

43. Provide support for different testing strategies using testing libraries like JUnit, Mockito and Cucumber.

44. Prevent concurrent submissions of the same code (Issue #19). Express.js has a rate limiting middleware for doing this.

45. Follow express security best practices.

46. The execution_node.js forks and executes a parameterized shell script. See the corresponding code. Sanity checking on the shell script parameters is required.

47. Right now, each test requires separate compilation and run. If all tests belong to JUnit, we can use/refactor JUnit and JUnitParams to complete testing much more quickly. See JUnit runners, theories and results classes/packages. They are enormously helpful in cutting down the Java testing time by at least 80%.

48. Refactor the code base to be microservices architecture compliant. Before refactoring the architecture, select appropriate ER model for the database and move to a lighter, scalable database than MySQL. PostgreSQL may be an appropriate DB for persistence with redis being the cache. While doing this, we need to keep in mind the need to distribute the execution node to the student. We also need to think about the possibility of refactoring load balancer into two sub-components: one is load balancer which can be taken care of by HAProxy and other is the controller which is application specific.

49. Private scoreboard - give instructor an option to hide the scoreboard. In case of hidden scoreboard, summary statistical view can be provided for students.

50. Provide evaluation logs containing details like <submissionID, studentID, lab, commit hash, language, time, testcase status array, testcase marks array> for each lab to the instructor. We can have a submissionID generated in mainserver and stored as primary key for the evaluation log table. Log this information in a DB table and provide an export option. Also provide simple statistics for each lab.

51. Decouple front-end web application from database. All database interactions happen through load balancer. At that point, front end website can go into a static content server / CDN.

52. The browser code of AutolabJS application reloads the index.html page completely for every new evaluation. This is a waste of network bandwidth and slows down the user experience. It is better to reuse the cached page of web browser. Check the options of express.static.

53. Make the distributed interactions be REST compliant. See express.json

54. Provide meaningful status messages for pending and ongoing evaluations. Provision for canceling pending or ongiong evaluations.
    There might be a need to provide visual, explicit feedback to the users about the existing evaluation queue. Knowing the queue status and queue size value would stop users from flooding the busy system with multiple requests. This endless loop scenario also motivates the need for provision to cancel an evaluation request.

55. separate browser code to another repo (see issue 170)

56. Refactor view part of web application using react and redux libraries. Use state pattern for properly utilizing socket.io-client state on the client side. At present the state of a long running socket.io connection is very buggy and unpredictable. See rxjs for a simple overview of react in JS.

57. Think about the applicability of state machine pattern for refactoring all the autolab components.

58. Make the C driver (Driver.c) use exception handling using setjmp() and longjmp() functions. For concrete implementation ideas, see Chapter 13: Exceptions, Object Oriented Programming with ANSI-C, Prof. Axel Tobias Schreiner. See issue #102

59. On-commit evaluations from command-line using one-step submission script.

60. A different, rich view of scoreboard for instructor. The instructor's scoreboard must contain links to code directory of student, commit hash, programming language used, time of submission.

61. Microbenchmarking for all the tests using benchmarking software like benchmark for CPP. (find equivalent ones for other languages)