Skip to content

Decision Log

Jump to bottom
P. Rouvilllain edited this page May 23, 2024 · 3 revisions

Here, we log the various decisions we have made about the MiniTwit development, deployment, and maintenance.

Choice of CI/CD platform

  • We have chosen GitHub actions
  • For our CI/CD setup, we wanted to use the tools we are already using to the furthest extent possible, to concentrate the number of platforms where our project is present, in the name of simplicity.
  • Since our version control is already facilitated on Github, it makes a lot of sense to locate the CI/CD pipeline in close proximity to the changing code.
  • Here, we have plenty of compute time, and the configuration is dead-simple.
  • Furthermore, by having our distributed version control and CI/CD pipeline on the same platform, we reap the benefits of having a close association between each change in code and the outcome of building and deploying the system with that change.
  • For example, we can implement an automated build-and-test procedure, that is triggered both when a Pull Request and when a new version is released.
  • Currently, we have a build-and-test procedure that is required to pass for a Pull Request to be approved.
  • Another major impact of using GitHub Actions is that now the task of building our application is carried within GitHub, and not on our VM. In theory, this should free up some processing power on the VM, because the only purpose of the VM is now to execute the application (besides keeping up to date with the latest version of the software).

Choice of Deployment Target and Virtualization

  • We have chosen to an Azure VM.
  • We are aware that Azure is a PaaS, but we will not take advantage of this, to the best of our ability.
    • We chose Azure since we were not able to successfully create accounts on DigitalOcean, while Azure very quickly granted us student credits.
  • An advantage of Docker is that SDK's are provided for many languages.
  • Our VM deployment script was relatively easy to write in C# because of this.
    • It should be noted that the deployment script is not currently fully functional, and needs additional work, as of 25th of March, 2024.

Framework Choice

We chose to base our refactoring on ASP.NET Core Razor pages for the following reasons:

  • Multiple group members are familiar with C# and ASP.NET
  • All group members are familiar with OOP
  • Razor Pages are relatively simple to implement and have a good separation of concerns
  • Since we are in the .NET world, we can widely expand our system later in the course; we are not constrained to a small framework, and as such, we won't have to change much to scale up the system.
  • ASP.NET comes with EF Core, which allowed us to use the database mostly without writing SQL-code, which made implementing the data layer a quick affair.
  • It is scalable performance-wise.
  • There is continuous development and improvements to .NET from Microsoft side.
  • There are lots of libraries (nuGet packages) to help implement new features.
  • It is open-source.
  • It uses C# which has a well known syntax, so it is a familiar experience.

Decisions taken during rewrite from Flask

  • We have decided to keep the current database, so we have all the current twits.
  • But, we are not going to support logging in to the users that are currently on the database.
    • The DB uses a format specific to Flask that is difficult to parse.
    • Furthermore supporting the current users would require supporting different hashing algorithms, creating a bespoke authentication system and hindering security.

Logging

First we tried to implement our logging with kibana and elastic search, which worked on our local machines. The problem with this was that we only had 1GB RAM on our web server, which wasn't enough to run elastic search. To resolve that issue we looked for other technologies like fluentd.

When facing implementing logging to a system several challenges come up, for example there are disparate log formats of different applications, logs that are scattered across multiple sources (data fragmentation), reliability concerns like network outages and routing different logs to different destinations based on the application or service.

In order to manage these difficulties we decided to use fluentd. Fluentd is an open source data collector which processes the data from various sources, making it easier to manage and analyze. The following aspects show why fluentd is so powerful:

  • Unified Logging Layer: Fluentd provides a unified platform for collecting, transforming, and routing logs from diverse sources, simplifying the log management process.

  • Data Agnosticism: Regardless of the data source---be it Prometheus metrics, Grafana dashboards, or application logs---Fluentd seamlessly collects and processes data in a unified format, fostering interoperability and ease of analysis.

  • Configurable Routing: With Fluentd, we could effortlessly configure log routing based on application or service, directing logs to specific destinations for centralized storage and analysis.

  • Flexibility: Fluentd's versatility enables us to send data from any source to any destination, empowering us to adapt to evolving logging requirements seamlessly.

Clone this wiki locally