This project is related to the course Cloud Technologies Course (PROG2005), Spring Semester 2023, taught at NTNU, Gjøvik. The project consists of a REST application in Golang that provides the client with the ability to retrieve information about developments related to renewable energy production for and across countries. The application uses an existing service and an own data-centric service to gather and expose data via endpoints. The service allows for notification registration using webhooks, and it is dockerized and deployed using an IaaS system.
NOTE: Firebase must be uploaded. Read more here
REST Web Services The REST web service used for this purpose are:
- REST Countries API (instance hosted for this course)
- Endpoint: http://129.241.150.113:8080/v3.1
- Documentation: http://129.241.150.113:8080/
- Renewable Energy Dataset (Authors: Hannah Ritchie, Max Roser and Pablo Rosado (2022) - "Energy". Published online at OurWorldInData.org. Retrieved from: https://ourworldindata.org/energy
- Endpoints
- Current endpoint
- History endpoint
- Notification Endpoint
- Status endpoint
- Default endpoint
- Testing
- Deployment
- Advanced functionality
- Design
- Further development
These are the endpoints for our REST web application:
/energy/v1/renewables/current
/energy/v1/renewables/history/
/energy/v1/notifications/
/energy/v1/status/
This endpoint retrieves the elements of the latest year currently available. The newest data in renewable-share-energy is from 2021, and is therefore the current year of this project.
Features of this endpoint:
- Search for country by name and country code.
- Add-on to get neighbouring countries.
- Cache for reducing amount of calls to countries API.
- Sorting of results.
The endpoint uses a file: "renewable-share-energy.csv" and REST Countries API, which is retrieved from: http://129.241.150.113:8080/v3.1. The file contains historical data from each countries' share of renewable sources.
Method: GET
Path: /energy/v1/renewables/current/{country?}{?neighbours=bool?}{sortbyvalue/sortalphabetically=bool?, descending=bool?}
Using no extra parameters will print all countries of the year=2021, to the client. The year found is based on the highest year found in the csv file.
{country?} is an optional parameter which could be passed to the API, which will print information about the country
as long as it is found. It could be a 3-letter country code, or country name.
{?neighbours=bool?} is an optional query parameter which will print information about the neighbouring countries of the
country passed. It therefore, dependent on the optional parameter: country.
{?sortbyvalue=bool?} is an optional query parameter which will sort the results by percentage.
{?sortalphabetically=bool?} is an optional query parameter which sorts the results alphabetically.
{?descending=bool?} is an optional query parameter which is dependent on the other sorting queries. When used it will sort descending instead.
The endpoint only supports GET requests.
Content type: application/json
Status codes:
- 200: Success, everything works as intended.
- 400: Bad request, error in the request. For example sent a number when it should have been a string.
- 404: Not found, for example did not find country code based on search.
- 500: Internal server error, error due to the server. For example, faulty file reading.
- 501: Not implemented, function is not implemented.
Body:
{
"Name": "country_name", (string)
"IsoCode": "country_code", (string)
"Year" "year_recorded", (int)
"Percentage": "percentage_of_renewables" (float64)
}
Request sent: /energy/v1/renewables/current/swe
Response:
{
"name": "Sweden",
"isoCode": "SWE",
"year": 2021,
"percentage": 50.924007
}Request sent: /energy/v1/renewables/current/norway?neighbours=true
Response:
[
{
"name": "Norway",
"isoCode": "NOR",
"year": 2021,
"percentage": 71.558365
},
{
"name": "Finland",
"isoCode": "FIN",
"year": 2021,
"percentage": 34.61129
},
{
"name": "Sweden",
"isoCode": "SWE",
"year": 2021,
"percentage": 50.924007
},
{
"name": "Russia",
"isoCode": "RUS",
"year": 2021,
"percentage": 6.6202893
}
]This endpoint retrieves all elements from renewable-share-energy. When no query is passed it will return the mean of all
data based on each country.
Functionality of history endpoint:
- Search for specific countries based on country code and name.
- Allows for searching for specific years.
- Allows for searching to, from and between specific years.
- Sort by percentage and alphabetically, both descending and ascending.
- Calculating the mean of a country.
The endpoint uses a file: "renewable-share-energy.csv" and REST Countries API, which is retrieved from: http://129.241.150.113:8080/v3.1. The file contains historical data from each countries' share of renewable sources.
REQUEST: GET
PATH: /energy/v1/renewables/history/{country?}{?begin=year?}{?end=year?}{?mean=bool?}{?sortbyvalue=bool?}
When you use this endpoint with no parameters or queries, it will print the mean of all historical entries for each country. The data is retrieved from renewable share energy.
{country?} is an optional parameter which could be passed to the API, which will print information about the country
as long as it is found. It could be a 3-letter country code, or country name.
{?begin=year?} is an optional query parameter used to filter results from a specific year.
{?end=year?} is an optional query parameter used to filter results to a specific year.
{?begin=year&end=year?} using both begin and end it will return results between the years written.
{?mean=bool?} is an optional query parameter, which will only work in tandem with country, begin or end. It
calculates the mean of the elements returned. This is done if no queries is presented.
{?sortbyvalue=bool?} is an optional query parameter which will sort the results by percentage.
{?sortalphabetically=bool?} is an optional query parameter which sorts the results alphabetically.
{?descending=bool?} is an optional query parameter which is dependent on the other sorting queries. When used it will sort descending instead.
The endpoint only supports GET requests.
Content type: application/json
Status codes:
- 200: Success, everything works as intended.
- 400: Bad request, error in the request. For example no country matching search parameter.
- 404: Not found, for example did not find country code based on search.
- 405: Method not allowed, writing invalid value in query. Example: ?query=notSupposedToBeLikeThis.
- 411: Length required, need more information to work.
- 500: Internal server error, error due to the server. For example, faulty file reading.
- 501: Not implemented, function is not implemented.
Body:
{
"Name": "country_name", (string)
"IsoCode": "country_code", (string)
"Year" "year_recorded", (int)
"Percentage": "percentage_of_renewables" (float64)
}
Request sent: /energy/v1/renewables/history/sverige?mean=true
Response:
[
{
"name": "Sweden",
"isoCode": "SWE",
"percentage": 33.970860684210535
}
]Request sent: /energy/v1/renewables/history/nor?begin=2011&end=2014&sortbyvalue=true
Response:
[
{
"name": "Norway",
"isoCode": "NOR",
"year": 2012,
"percentage": 70.095116
},
{
"name": "Norway",
"isoCode": "NOR",
"year": 2014,
"percentage": 68.88728
},
{
"name": "Norway",
"isoCode": "NOR",
"year": 2013,
"percentage": 67.50864
},
{
"name": "Norway",
"isoCode": "NOR",
"year": 2011,
"percentage": 66.30012
}
]To get notified by a given amount of calls a country has, register a webhook with this service.
Note that firebase must be setup! Read this section!
To the body make sure to add:
- the url that should be invoked
- the alpha code of the country that you want to be notified by
- the number of calls to be notified if the event is for calls.
- the type of event to be notified on. See the notification types here.
Provide the following details to get notifications to the given url. The standard way is that the user will receive a GET request for the given url in the body. Here is how you register a notification:
REQUEST: Post
PATH: "/energy/v1/notification"
BODY:
{
"url": "The given url for the webhook to call",
"country": "Alpha code of the country",
"calls": "Number of calls for notification"
"event": "Type of event"
}
The response should be 201 Created if all went well. See the error message for more details.
You should also the webhook ID in the body of the response. This ID is important, so save it for either deletion or retrieving details about it. Here is an example response:
{
"webhook_id": "OIdksUDwveiwe"
}To delete a webhook, send a DELETE request to the following endpoint, including the ID of the webhook in the URL:
REQUEST: DELETE
PATH: /energy/v1/notifications/{webhook_id}
Look at the status code for how the request for deletion went. If the status was:
- 400: Please make sure that you added an ID the url.
- 200: Webhook was either found and deleted, or not found (so nothing happened)
- 500: Internal error while trying to delete the webhook. See the status endpoint to check if all services are running
To get only information for a single given notification, use the id in the request:
REQUEST: GET
PATH: /energy/v1/notifications/{webhook_id}
Look at the status code and message if no webhook was received.
If there is a webhook with the given ID, the response could look like this:
{
"webhook_id": "ID_of_the_webhook",
"url": "Url_of_the_registration",
"country": "Alpha_code_of_the_country",
"calls": "The_amount_of_calls_that_needs_to_be_for_invoking",
"event" : "The event type of the notification",
"created_timestamp": "Server_timestamp_when_the_webhook_was_created",
"invocations": "The_amount_of_times_the_country_with_the_given_alpha_code_has_been_invoked"
}To get all the notifications that are stored in the register:
REQUEST: GET
PATH: /energy/v1/notifications/
Should return a list of all webhooks. Could also be empty if non are registered yet. Expected response would look like this;
[
{
"webhook_id": "ID_of_the_webhook",
"url": "Url_of_the_registration",
"country": "Alpha_code_of_the_country",
"calls": "The_amount_of_calls_that_needs_to_be_for_invoking",
"event": "The_type_of_event",
"created_timestamp": "Server_timestamp_when_the_webhook_was_created",
"invocations": "The_amount_of_times_the_country_with_the_given_alpha_code_has_been_invoked"
},
...
]This service offers three types of of events:
- PURGE:
- Description: Notification if the service had to purge webhooks (due to stepping over the limit)
- Does it delete itself after invocation? No, the notification is saved.
- Example of registration:
REQUEST: Post PATH: "/energy/v1/notification" BODY: { "url": "https://webhook.site/url-stuff", "event": "purge" }
- CALLS:
- Description: When the number of invocations is dividable by the calls number
- Does it delete itself after invocation? No, the notification is saved.
- Example of registration:
REQUEST: Post PATH: "/energy/v1/notification" BODY: { "url": "https://webhook.site/url-stuff", "country": "NOR", "calls": 4, "event": "calls" }
- COUNTRY_DOWN:
- Description: If the country API goes down, a notification is sent. Only happens if the status endpoint gives other status code for the country API then 200
- Does it delete itself after invocation? Yes, once invoked, a new notification is needed.
- Example of registration:
REQUEST: Post PATH: "/energy/v1/notification" BODY: { "url": "https://webhook.site/url-stuff", "event": "country_down" }
Here is an example of the JSON response you will receive when a notification is triggered based on the calls event:
{
"webhook_id": "32b184e5bc9e9bee7fdff1362dc2e05bc7174290a4cc3622cd39f5b1803c97e6",
"url": "https://webhook.site/sample_url",
"country": "NOR",
"calls": 2,
"event": "CALLS",
"invocations": 38,
"message": "Notification triggered: 38 invocations made to NOR endpoint."
}When the user adds a notification, a method called PurgeWebhooks is called. It checks if the amount of webhooks is now over the limit. If it is, then it starts removing the oldest notifications. It only removes enough webhooks so that the total amount of notifications are stored is under the limit. By default, the total amount of webhooks allowed is 40. This could also be changed in the constants file.
- Did not choose to delete webhooks that has the least amount of invocations, because new notifications would be deleted.
- Improvements could be to update the creation time, whenever information has changed. Disregarded this due to conflict of naming: creation does not imply last updated, and also another field to keep track on would lead to unnecessary storage of data.
Whenever there is a request to the third party api, restcounties, we increment all the webhooks for that country with one. In the same function we notify the user, if the condition of being notified is met.
Using the firebase cloud storage called: Firestore. The application uses firestore to store webhooks in form of documents. See document databases for more information on how this works. The technical aspects for getting this to work is;
- Having a firestore credentials file in the root folder of the project.
- The credentials file MUST be called cloud-assignment-2.json
- Manually created two collections called: test_collection and webhooks
Note: changes might lead to errors, so don't.
This is also located in the constant code:
const (
....
FIRESTORE_COLLECTION = "webhooks"
FIRESTORE_COLLECTION_TEST = "test_collection"
FIREBASE_CREDENTIALS_FILE = "cloud-assignment-2.json"
...
)The status endpoint provides the availability of all individual services this service depends on. The reporting occurs based on status codes returned by the dependent services. The status interface further provides information about the number of registered webhooks, and the uptime of the service. It also provides the total memory usage of the computer in use.
Method: GET
Path: /energy/v1/status/
Content type: application/json
Status codes
- 200: Everything is OK.
- 404: Not found.
- 500: Internal server error.
Status content
- countries_api: the http status code for the "REST Countries API".
- notification_db: the http status code for "Notification DB" in Firebase.
- webhooks: the number of registered webhooks.
- version: set to "v1".
- uptime: the time since the last service restart.
Request: /status
Response:
{
"countries_api": "http status code for restcountries API",
"notification_db": "http status code for notification DB in Firebase",
"webhooks": "amount of registered webhooks",
"version": "v1",
"uptime": "time elapsed from the last service restart"
"total_memory_usage": "percent of total memory usage on the user's computer"
}
Note: "some value" indicates placeholders for values to be populated by the service.
An example response is provided underneath.
Example response:
{
"countries_api": 200,
"notification_db": 200,
"webhooks": 2,
"version": "v1",
"uptime": "10 seconds",
"total_memory_usage": "78%"
}
This endpoint is the server's root path level. It does not provide any functionality, but assists the user to navigate in the server. The HTML file in linked up with a css file in order to provide a more clean look to the page, with the endpoints being displayed in an organized and easy-to-use format. It is possible to press the different endpoints to navigate to their respective endpoints.
REQUEST: GET
PATH: /energy/
We strive to maintain a high level of quality in our tests to ensure the reliability and stability of our application. We use a combination of inbuilt Go testing and the testing framework, Testify, to ensure that our code is reliable and bug-free. Testify provides several convenient assertion functions that allow us to test our code with ease. We also use the built-in testing package of Go, which provides a robust testing infrastructure.
To ensure that our tests cover as much of our codebase as possible, we make use of coverage tools such as:
go test -cover, which shows us the percentage of code covered by our tests. By maintaining a high level of code
coverage, we can be confident in the quality of our codebase, and we can easily catch any bugs that may arise
during development.
To run all endpoint tests write the following command in root folder:
go test .\internal\webserver\handlers
There is created a test class for the current endpoint.
To use the test, print into command line when in root project folder:
go test .\internal\webserver\handlers\current_test.go
There is a test class for the history endpoint, which covers most of the history endpoints' functions.
To use the test, print into command line when in root project folder:
go test .\internal\webserver\handlers\history_test.go
The notification test are highly coupled with the Firebase test. Therefore are the notification test only to check that the endpoint works as it is supposed to. This means that it may be lacking. However, the Firebase test should have no issue if Firestore is correctly setup. From this if:
- FIRESTORE && NOTIFICATION ENDPOINT TEST FAIL -> Most likely just incorrectly setup the firestore
- ONLY NOTIFICATION ENDPOINT FAIL -> Logical error in the code in notification.go
To test the firebase methods only:
go test ./db
There is created a test class for the status endpoint.
To use the test, print into command line when in root project folder:
go test .\internal\webserver\handlers\status_test.go
There is created a test class for the default endpoint.
To use the test, print into command line when in root project folder:
go test .\internal\webserver\handlers\default_test.go
This service is deployed with OpenStack. OpenStack is a IaaS where the user define what resources is needed. It vitalizes resources to serve all end users. More information here: Openstack Link
The service is deployed with openstack.
Access it with this floating IP:
http://10.212.169.162:8000/energy/v1/status/
Note: In the case of self-hosting, use the floating IP of the instance.
This service has the following resources predefined:
- Ubuntu Server 22.04 LTS (Jammy Jellyfish) amd64
- gx1.1c1r flavor
Security group prevents all communication with the server, but this is allowed:
- Allows to any ICMP package (Ping is allowed)
- SSH (Port 22)
- Http (Port 8000)
To access our service you need be connected to the NTNU network. This could also use the Cisco VPN to connect to the campus network. More information about the VPN here!
Docker is a set of platform as a service (PaaS) products that use OS-level virtualization to deliver software in packages called containers. This project used this docs for setting up docker on the OpenStack server.
Note that the project uses both a Docker file and a docker compose file. The docker file contains introductions for building the Docker Image. By defining a base image, additional packages and other versions, the Docker image can be created more deterministic. The base image that we use is golang:1.18. The Docker File also gets all packages from the go project.
Read more about how dockerfile works here
Docker Compose is for running multi-container Docker application. However, this project uses it to define volumes for the credentials file for firebase. Also the renewable energy .csv file should also stay in a volume.
The following steps are
- Connect to the NTNU Campus network (Via VPN or direct connection)
- Create an OpenStack Instance has the same flavor and OS that has been specified in this README
- Create and add Security Policy to the Instance with
- SSH to port 22 (For logging on to the instance)
- Ingress at port 8000 (For accessing the service)
- Create and add SSH key to the Instance
- Store the
.pemfile for logging in to the server
- Store the
- Login the server using the floating IP and the
.pemfile with:ssh -i ./name-of-ssh-key.pem ubuntu@"YOUR_FLOATING_IP"- Common errors: Not correct permissions to the
.pemfile or that the ssh key has not been set to the instance - Other errors is due to not correctly deploying an instance, see OpenStack introduction docs here
- Common errors: Not correct permissions to the
- Installed docker: Docker installation manual for ubuntu here.
- Clone the repo to the machine using
git clone - Use the
scplinux command for adding the firestore credential file:- Secure copy command article
- See the section on Storing notifications for how Firestore should be setup
- File must be named:
./cloud-assignment-2.json - File must be moved inside the repo at the root of the project
- OPTIONAL Set docker to the group of sodu privileges. The rest of the steps assumes docker can be used without sudo. By default, docker needs sudo privileges to run. Can also use
sudo docker ....when using docker commands. - Build and deploy with this command. Uses the compose file. Also detaches form the :
docker compose up -d - Verify that compose the service has been deployed by using the docker command for checking on the service:
docker ps -a
- Have go installed on the local machine. See download versions here (use go.1.18)
- Clone the repo.
- Run the project by cd into the project folder, then run:
go run ./cmd/main.go
- See logs for the port of running service. (Usually port 8080)
- Access the service with local host here:
http://localhost:PORT_NUMBER/energy/v1/status/
This assignment introduced the following advanced tasks:
Description: Implement purging of cached information for requests older than a given number of hours/days.
We have implemented a solution that functions using maps which stores countries which have been collected from the countryAPI earlier. The key is the name of the country and its value is country structs. It is mostly used for the neighbour query in the current endpoint. When searching for a country/ies, the country/ies will be cached for a set period of time, which we have set to 600 hours (equal to 25 days). This is done to ensure an updated cache. The result is less frequent requests for the API and shorter response time. When searching for a country/ies, the country/ies will be cached for a set period of time. This is done to ensure an updated cache. This result in less frequent requests for the API and shorter response time.
With ?information=true on any given endpoint, for more sample queries for the endpoint. Default handler will redirect with this option selected. Each query example follows the flowing structure:
{
Title: "Title of the query",
Example: "Use: PATH_TO_ENDPOINT",
Description: "Description of the query",
}The notification supports different types of events. There are currently 3 types of events. Read more about them here
When the number of notifications are over the max limit defined, webhook will be deleted. Read more here
Description: Extend {?country} to support country name (e.g., norway) as input. Read more here
Description: Selective use of only begin or end as single parameter (e.g., ?begin=1980 only consider data from 1980 onwards; ?end=1980, values from the first time entry until 1980 only). Read more here
Description: Extend the history for all countries with a time constraint. Where {?begin=year&end=year?} is specified (e.g., ?begin=1960&end=1970), only calculate mean values for these years (not for all years). Read more here
Description: Additional optional parameter {?sortByValue=bool?} to support sorting of output by percentage value (e.g., ?sortByValue=true). Read more here
Both current and history endpoint has the functionality of searching by country code and also name, which is an advanced
functionality. We have implemented another bonus functionality, which searches the API and if it does not find a country
in the csv file, it will use the API: http://129.241.150.113:8080/v3.1/name/.
It will then search for any type of name in country body, which could be common, official and nativeName.
This allows for searches of /history/Kongeriket Norge, which will return information about norway.
This feature will increase the amount of calls to API, which is sometimes unnecessary. If a user writes gibberish into our API, it will search the country API. However, using the API a user may even search the countries native name and receive the correct country.
Request: /current/España
Response:
[
{
"name": "Spain",
"isoCode": "ESP",
"year": 2021,
"percentage": 22.341663
}
]We have also created a wiki for this project which includes more information about the assignment, like for example an overview of the applications' use-case examples and the group dynamic during the assignment.
Throughout the implementation of this application, the focus points on the design has been loose coupling, high cohesion and modularity as close to Golang convention as possible. This has been done through constants, different files for handlers and generic functions.
The project structure was created with the goal of responsibility driven design, and to minimize code duplication overall.
The endpoint-handlers got one file each, and are all located in the "handlers" package.
In order to limit API-requests, when countries are requested all borders from the country is retrieved and for each border request the country. By doing this the API workload can get large. The API-server side's workload is reduced. In this way the REST-principles are met.
These are further improvements we did not have time to resolve.
Let the user get extra functionality based on their user role, for example an administrator. An administrator would have certain privileges to data related to the server health, examples are provided below:
- Response time: Measure the time it takes for the service to respond to requests.
- Error rate: An idea of the number of errors that occur in the service.
- Request count: The number of requests made to the service.
This could be solved through making requests with a HEAD field including a passphrase or password. A user without the credentials in the HEAD field would not be able to access the endpoint that offers the privileged functionality since the user would not be authenticated.
We could have put Go files into folders to increase cohesion and make the structure more clean, however the structure we have now has a relatively high cohesion as the handler folders only works with handler functions. While the utility folder only works with helper functions used in the handlers.
The time writing this, the test coverage lies around 70% of the lines in the tested packages. However, some packages are not tested yet. For example, we could have implemented another test for a stubbed countryAPI. However, we didn't meet the time requirement to implement this. On the other hand the main functionality of all packages is tested in varying degrees, and the other functionality is indirectly tested as well.
- Use a middleware to set the content-type header for all response.
- Implement Gorilla Mux to define URL routes and extract variables from them instead of doing it manually.