Table of Contents generated with DocToc
- Description of the tool
- Installation
- Authentification
- Python library
- API endpoints
Here are the main features:
- NoSQL database: MongoDB
- Node.js
- express (to build web application and API)
- express-session (to handle user session)
- cookie-parser (to handle cookies)
- mongodb and connect-mongo (to access the database)
- mongoose (to easily make queries on the database)
- bcrypt (to encrypt passwords)
- md5 (to encrypt tokens)
- uuid (to generate Universally Unique Identifier)
- multer (to upload files on the server)
- archiver (to zip the files)
- node-stream-zip (to unzip the files)
- diskspace (to have information about the disk usage of the server)
- get-folder-size (to know the size of a folder)
- pm2 (for production, to restart automatically the application if it crashes)
- Postman (best GUI to make HTTP requests, useful to try queries on the API)
- Robo 3T (GUI for MongoDB)
To deploy the application, you can either use Docker and Ansible (automated deployment), or install everything manually.
Coming soon...
On the server:
- Open a terminal and run
sudo service mongod start
- Open a Mongo Client in the terminal
mongo
- Create the database
use ASTR
- We need to insert a document to complete the creation of the database. Let's insert an empty document in the collection "uselesscollection"
db.uselesscollection.insert({})
- Now, check that the database is in the list of existing dbs
show dbs
curl -sL https://deb.nodesource.com/setup_9.x | sudo -E bash -
sudo apt-get install -y nodejs
git clone url_of_astr_repository
- In your terminal, move to the folder of the repository
- At the root of the folder run
npm install
-
If an error occured, try with
sudo -
It will install all the Node.js modules used in the application (listed in package.json)
-
Install pm2 module
npm install pm2 -g
- At the root of the folder run
npm run prod
↪️ This command starts the application with pm2 at port 8000.
- To use a different port, launch the application as follow
# for production
npm run prod -- -- 3000
- To stop the server in production mode
npm stop
# OR
pm2 delete {pm2_processs_id}
From your personal computer, open the website (serverIP:8000)
- Click on Login
- Click on Register an Account
- Fulfill the form to create your account (it will create a simple user without any permission)
From the server, open a terminal
- Open the mongoDB client
mongo
- Switch to ASTR database
use ASTR
- Update your account with full privileges
db.users.update({"email": "yourEmail"}, {"$set": {"master": true, "write_permission": true}})
- That's it! You are now a "Master", that means you can modify directly users permissions on the website
By default, the archives are stored in a folder named "archives" at the root of ASTR application. If you wish, you can configure a different path.
The path must be absolute and without "/" at the end (example: /home/john.doe/Documents/astr_archives). If the last folder of your path doesn't exist yet, it will be created automatically.
To configure the new path:
- execute the following commands from the server:
mongo
> use ASTR
> db.applications.findOneAndUpdate({}, {$set: {archivesPath: "/your_new_path"}})
- restart ASTR
cd astr
npm stop
npm run prod
- If you started the application with
npm run prod, you can monitor it by running
pm2 monit
- You will be able to see some informations like the application logs, the CPU utilization, the number of restarts, etc.
Some requests to the API require authentification. A website user doesn't need to worry about it because it is completely transparent and handled with cookies. But a script user will have to authentificate to do some actions.
To verify that the user has the required authorizations, he has to authentificate in the request header with his email and one of his tokens, using Basic authentication.
Authorization: Basic email:token
If the user doesn't authentificate or give a wrong token, an 401 error (Unauthorized) will be return.
An example with curl:
curl -X DELETE \
-u john.doe@mail.com:d2147e39-8b6e-4c7b-b4ca-f93529dfbbd1 \
http://hostname:8000/api/archives/id/5b19442c5dd23f39e6f5e6d8
In the Python library, everything is already handled by client.py.
The user only needs to configure his environment variables (cf. configuration)
Tokens are uuid (v4): string of 32 random hexadecimal digits. They are stored encrypted in the database using MD5 hash function. MD5 is known to be vulnerable, but it is perfectly safe to use it on long strings of random characters. Therefore, there is no need to overload the server CPU with more complex hashing algorithms.
Users have two types of tokens:
- session-tokens: for website usage. A new token is created on login and is deleted on logout. It is stored in the client cookies and used for requests with authentication. This process is completely transparent for the user.
- persistent-tokens: for script usage. The user can create as many persistent-tokens as he wants on the profile page. All of them have a name and an expiration date of one year. The user must store these tokens in local files on his PC. In fact, they are stored encrypted on the database, so it won't be possible to access their original form after creation.
Tokens expiration dates are checked when the user logs in. If the date is passed, then the token is removed from the list and therefore can no longer be used.
Libastr is a Python3 library designed to ease python scripting with A.S.T.R. API. It includes multiple features like retrieving, downloading, uploading archives.
Find all the information about it in the dedicated repository: lib-python-astr
- /api/archives
- GET: Returns the list of all archives (sorted by creation date in descending order)
- POST: Returns the list of archives that match with the parameters given in the body request (sorted by creation date in descending order)
- /api/archives/page/:page/:resultPerPage
- POST: Returns the list of archives that match with the parameters given in the body request, with pagination (sorted by creation date in descending order)
- /api/archives/add
- POST: Add a new archive in the DB in function of the parameters given in the body request (user must have write permission)
- /api/archives/id/:id
- GET: Returns the archive with the associated ID
- POST: Update the archive with the associated ID in function of the parameters given in the body request (only the date, the comments, and the descriptors values can be updated) (user must be master or owner of the archive)
- DELETE: Delete the archive with the associated ID (user must be master or owner of the archive)
- /api/archives/authors
- GET: Returns the list of archive authors (that added at least one archive)
- /api/archives/categories
- GET: Returns the list of archive categories (used at least by one archive)
- /api/archives/descriptors
- GET: Returns the list of descriptors (used at least by one archive)
- /api/archives/descriptors/:category
- GET: Returns the list of descriptors of the associated archive category (used at least by one archive)
- /api/archives/options/:descriptorName
- GET: Returns the options of the associated descriptor (used at least one time)
- /api/archives/changeArchiveCategoryName
- POST: Change the category name of all the archives matched by {category: previousName} (body contains previousName and newName) (user must be master)
- /api/archives/addDescriptor
- POST: Push a new descriptor in all archives matched by the archive category (body contains category and descriptor: {name, value}) (user must be master)
- /api/archives/changeDescriptorName
- POST: Change the name of the matched descriptors in all archives matched by the archive category (body contains category, previousName and newName) (user must be master)
- /api/archives/withoutZip
- GET: Returns the list of all archives that are missing in the folder "archives" (to delete them)
- /api/archives/YAMLformat/id/:id
- GET: Returns the archive with the associated ID in a YAML format, to store it in the zip
- /api/archives/cleanArchivesFolder
- GET: Clean the archives folder and returns the deleted files (delete files/folders not created today, that don't have the zip extension and a filename different than 24 hexadecimal digits). Executed automatically once a day.
- /api/categories
- GET: Returns the list of all archive categories
- POST: Add a new archive category in the DB in function of the parameters given in the body request (user must be master)
- /api/categories/id/:id
- GET: Returns the archive category with the associated ID
- POST: Update the archive category with the associated ID in function of the parameters given in the body request (user must be master)
- DELETE: Delete the archive category with the associated ID (user must be master)
- /api/categories/name/:name
- GET: Returns the archive category with the associated name
- /api/categories/options/:category/:descriptorName
- GET: Returns the options of a descriptor
- /api/categories/links/:category
- GET: Returns the links of an archive category
- /api/search
- GET: Returns the list of all saved searches
- POST: Add a new search in the DB in function of the parameters given in the body request (user must use authentification)
- /api/search/id/:id
- GET: Returns the search with the associated ID
- DELETE: Delete the search with the associated ID (user must be the owner of the search)
- /api/user
- GET: Returns the list of all the users
- POST: Used for connection if the body contains logemail and logpassword; or for adding a new user if the body contains email, firstname, lastname, password and passwordConf
- /api/user/master
- GET: Returns the list of all the masters
- /api/user/id/:id
- GET: Returns the user with the associated ID
- POST: Update the user with the associated ID in function of the parameters given in the body request (only the variable write_permission and master can be modified) (user must be master)
- DELETE: Delete the user with the associated ID (user must be master)
- /api/user/email/:email
- GET: Returns the user with the associated email
- /api/user/profile
- GET: Returns the information about the user logged in the machine
- /api/user/logout
- GET: Log out the user logged in the machine
- /api/user/newToken/:type/:name
- GET: Generate a new token for the user, returns it and store it encrypted in the database (type can be 'session' or 'persistent')
- /api/user/deleteToken/:id
- DELETE: Delete the token with the associated ID
- /api/upload
- POST: Upload files to the server in a ZIP. The name of the zip is the ID of the archive (user must have write permission)
- /api/upload/replace-zip
- POST: Replace zip with a new one (user must have write permission)
- /api/download/id/:id
- GET: Download the zip of the archive with the associated ID
- /api/download/multiple
- POST: Download a ZIP containing multiple archives. The archive IDs to download are passed in the body request.
- /api/download/files
- GET: Returns the list of files in archives folder
- /api
- GET: Returns information about the application (name, version, creation date, lastBootUptime)
- /api/change-app-name
- POST: Change the name of the application (to allow using a custom name) (user must be master)
- /api/stats/archiving-frequency
- GET: Returns a dictionnary with the number of archives uploaded per month
- /api/stats/disk-usage
- GET: Returns a dictionnary with the disk usage information