The codebase of CLEO
- Postgres, postgis.
- Python >=2.7, >=3.5
- Redis
- Elasticsearch 5.x
On Python 3.6 you will need to install the python3.6-tk
package for
tkinter
support.
sudo apt-get install python3.6-tk
Install NodeJS:
https://nodejs.org/en/download/
Then run:
yarn install
sudo apt-get install libgdal-dev
sudo su -l postgres
createdb -T template0 -E utf-8 -l en_US.UTF-8 -O postgres muses
psql -c 'create extension postgis;' -d muses
./manage.py migrate
Database engine shall be django.contrib.gis.db.backends.postgis.
DATABASES = {
'default': {
'ENGINE': 'django.contrib.gis.db.backends.postgis',
'NAME': 'muses',
'USER': 'whoever',
'PASSWORD': 'whatever',
}
}
Install
sudo apt install redis-server
Allow connections
- Edit the redis conf file
sudo nano /etc/redis/redis.conf
file. - Find line
bind 127.0.0.1
and change it tobind 0.0.0.0
- Save the redis.conf file.
Restart
sudo systemctl restart redis
or
/etc/init.d/redis-server restart
Test
redis-cli ping
The expected output is PONG
.
Either install it or run with Docker.
docker pull docker.elastic.co/elasticsearch/elasticsearch:5.5.3
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "xpack.security.enabled=false" docker.elastic.co/elasticsearch/elasticsearch:5.5.3
At the moment, data is imported from the following data sources:
- www.rmo.nl
- brooklynmuseum.org
- thewalters.org
- metmuseum.org
To obtain a CSRF token make use of /csrftoken/ endpoint.
Sample request:
POST http://localhost:8000/csrftoken/
Or with cURL:
curl -X POST -i http://localhost:8000/csrftoken/
Sample response:
{"token": "bsHtEMn9K8rqWSm1ql5hz5BcBNR3DuFJRgyc6VVE4c0Gh7PzjaEJap9FErsgxkBz"}
The description of the Artificial Intelligence (AI) or algorithm that was created for Cleo can be found at https://cleo.aincient.org/pages/en/ai/
There are two lists of synonyms for Elasticsearch:
- English: implementation/synonyms/raw/en.txt
- Dutch: implementation/synonyms/raw/nl.txt
Both of them are not in format accepted by Elasticsearch. We need to do some tiny transformation in order to make it acceptable.
Example of the English synonyms:
word_id;preferred_EN;variant1;variant2;variant3;variant4;variant5;variant6;variant7
1;Anatolia;anatolia;anatolie;anatolien;;;;
2;Assyria;assyria;assyrie;assyrien;;;;
3;Babylonia;babylonia;babylonie;babylonien;;;;
4;Byblos;;;;;;;
5;Crocodilopolis;;;;;;;
6;Greek-Roman;graeco-roman;greco-roman;;;;;
7;Herodote;herodotos;;;;;;
8;Horapollo;horapollon;;;;;;
9;Isis-aphrodite;;;;;;;
10;Manetho;manethon;manethos;;;;;
11;Maya;;;;;;;
12;Nefertiti;nefertete;nofretete;;;;;
13;Oxyrhynchus;oxyrhynchos;behnasa;el-Bahnasa;bahnasa;;;
14;pharaoh;pharao;;;;;;
15;Plutarchus;plutarchos;;;;;;
16;Punt;;;;;;;
17;Rosetta;Rosette;el rashid;el-rashid;al-rashid;;;
18;Saqqara;saqqarah;saqqareh;sakkarah;;;;
19;Serapeum;serapaeum;serapeion;;;;;
20;Sinuhe;sinouhe;;;;;;
21;Taffeh;taffah;taffa;tafa;taphis;;;
22;Thebes;;;;;;;
23;Teye;tiy;tiye;;;;;
24;Wenamun;wen-amon;wenamoen;;;;;
What is made of it:
anatolia, anatolia, anatolie, anatolien
assyria, assyria, assyrie, assyrien
babylonia, babylonia, babylonie, babylonien
greek-roman, graeco-roman, greco-roman
herodote, herodotos
horapollo, horapollon
manetho, manethon, manethos
nefertiti, nefertete, nofretete
oxyrhynchus, oxyrhynchos, behnasa, el-bahnasa, bahnasa
pharaoh, pharao
plutarchus, plutarchos
rosetta, rosette, el rashid, el-rashid, al-rashid
saqqara, saqqarah, saqqareh, sakkarah
serapeum, serapaeum, serapeion
sinuhe, sinouhe
taffeh, taffah, taffa, tafa, taphis
teye, tiy, tiye
wenamun, wen-amon, wenamoen
In detail:
- All one-word lines are being removed
- All words are lower-cased.
- All empty elements are removed.
- First row is removed.
- First element of each row (number) is removed.
Note
Do not commit synonyms files if anyhow the format has been changed (see the example of the synonyms file if anything (format, delimiter, etc) has been changed.
If no geo-location could be discovered, the values for latitude and longitude are set respectively to "-90.0" and "-180.0".
In order to log in, you should make a POST request with the following information to the /rest-auth/login/ endpoint.
Sample request:
POST http://localhost:8000/rest-auth/login/
{
"usernname": "admin",
"password": "test"
}
Or if we use cURL:
curl -X POST -H 'Content-Type: application/json' -i http://localhost:8000/rest-auth/login/ --data '{
"username": "admin",
"email": "",
"password": "test"
}'
Sample response:
HTTP 200 OK
Allow: POST, OPTIONS
Content-Type: application/json
Vary: Accept
{
"key": "144bae5d5e3f00bef13c92a5d33a327d7bc37f0a"
}
It's as simple as sending a GET request to the /rest-auth/logout/ endpoint.
Sample request:
GET http://localhost:8000/rest-auth/logout/
Sample response:
{"detail": "Successfully logged out."}
It's as simple as sending a GET request to the /rest-auth/user/ endpoint.
Sample request:
GET http://localhost:8000/rest-auth/user/
Sample response:
{
"pk": 3,
"username": "admin",
"email": "admin@localhost",
"first_name": "Admin",
"last_name": "Localhost",
"account_settings": {
"language": "nl"
}
}
It's a POST request to the /rest-auth/user/ endpoint.
Sample request:
POST http://localhost:8000/rest-auth/user/
{
"pk": 3,
"username": "admin",
"email": "admin2@localhost",
"first_name": "Admin",
"last_name": "Localhost",
"account_settings": {
"language": "en"
}
}
Sample response:
HTTP 200 OK
Allow: GET, PUT, PATCH, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept
{
"pk": 3,
"username": "admin",
"email": "admin2@localhost",
"first_name": "Admin",
"last_name": "Localhost",
"account_settings": {
"language": "en"
}
}
Post request to /rest-auth/registration/.
Sample request:
POST http://localhost:8000/rest-auth/registration/
{
"username": "test4",
"email": "test4@localhost",
"password1": "test1234",
"password2": "test1234"
}
Sample response:
HTTP 201 Created
Allow: POST, OPTIONS
Content-Type: application/json
Vary: Accept
{
"key": "eb7698d526bfa06bbba59cf747ed1c23f78866cd"
}
For non-authenticated users who forgot their password.
Post request to /rest-auth/password/reset/. If there's an account registered with the given e-mail address, a password reset e-mail would be sent to the latter for the confirmation.
Sample request:
POST http://localhost:8000/rest-auth/password/reset/
{
"email": "test4@localhost"
}
Sample response:
HTTP 200 OK
Allow: POST, OPTIONS
Content-Type: application/json
Vary: Accept
{
"detail": "Password reset e-mail has been sent."
}
Sample e-mail contents:
http://localhost:8000/rest-auth/password/reset/confirm/?uidb64=xxx&token=4xr-xxx
uidb64: xxx
token: 4xr-xxx
Make a POST request to the endpoint specified /rest-auth/password/reset/confirm/ providing the following fields:
- new_password1: Your new password
- new_password2: Repeat your new password
- uid: The value of uid64 you got in a confirmation e-mail.
- token: The value of token your got in a confirmation e-mail.
Sample request
POST http://localhost:8000/rest-auth/password/reset/confirm/?uidb64=xxx&token=4xr-xxxx
{
"new_password1": "newpass",
"new_password2": "newpass",
"uid": "xxx",
"token": "4xr-xxxxxx"
}
Sample response:
HTTP 200 OK
Allow: POST, OPTIONS
Content-Type: application/json
Vary: Accept
{
"detail": "Password has been reset with the new password."
}
For authenticated users who desire to change their current password.
Make a POST request to the /rest-auth/password/change/ endpoint.
Sample request
POST http://localhost:8000/rest-auth/password/change/
{
"new_password1": "newpass2",
"new_password2": "newpass2"
}
Sample response:
HTTP 200 OK
Allow: POST, OPTIONS
Content-Type: application/json
Vary: Accept
{
"detail": "New password has been saved."
}
Sample request
POST http://localhost:8000/account/usercollectionitemfavourites/
{
"collection_item": 36522
}
Sample response
HTTP 201 Created
Allow: GET, POST, HEAD, OPTIONS
Content-Type: application/json
Location: http://localhost:8000/account/usercollectionitemfavourites/10/
Vary: Accept
{
"url": "http://localhost:8000/account/usercollectionitemfavourites/10/",
"id": 10,
"user": 3,
"collection_item": 36522
}
Exceptions
If you try to add the same item to favourites more than once, you will get the following response:
HTTP 400 Bad Request
Allow: GET, POST, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept
{
"detail": "You have already added that item as a favourite."
}
To remove item from favourite, send a DELETE request to the item detail endpoint /account/usercollectionitemfavourites/{item-id}/:
Sample request
DELETE http://localhost:8000/account/usercollectionitemfavourites/1/
Sample response
HTTP 204 No Content
Allow: GET, PUT, PATCH, DELETE, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept
Get /account/usercollectionitemfavourites/show_indexes/.
Sample request
GET http://localhost:8000/account/usercollectionitemfavourites/show_indexes/
To export all favourites into CSV format, the /account/usercollectionitemfavourites/show_indexes/ endpoint is used.
Sample request
GET http://localhost:8000/account/usercollectionitemfavourites/export_all/
To export all favourites into excel format, the /account/usercollectionitemfavourites/show_indexes/ endpoint is used.
Sample request
GET http://localhost:8000/account/usercollectionitemfavourites/export_all/?docformat=xlsx
Endpoint: /api/collectionitem/{object-id}/download/
Sample request
http://localhost:3000/api/collectionitem/36534/download/
Endpoint: /api/collectionitem/{object-id}/download/
Sample request
http://localhost:3000/api/collectionitem/36534/download/?docformat=xlsx
We have a endpoint for user uploaded user images, which are used to find similar items.
Sample request
POST http://localhost:8000/account/usersearchimages/
{
"image": "base64:sdhfdsfjdskljfds"
}
Sample response
HTTP 201 Created
Allow: GET, POST, HEAD, OPTIONS
Content-Type: application/json
Location: http://localhost:8000/account/usersearchimages/2/
Vary: Accept
{
"url": "http://localhost:8000/account/usersearchimages/2/",
"id": 2,
"user": 3,
"image": "http://localhost:8000/media/image1.jpg",
"created": "2018-06-20",
"updated": "2018-06-20"
}
To run the frontend use the following code
yarn start
Go to http://localhost:3000/search/
To build the production version run the following command
yarn build
Project is covered with tests.
./runtests.py
It's assumed that you have all the requirements installed. If not, first install the test requirements:
pip install -r implementation/requirements/test.txt
Keep the following hierarchy.
=====
title
=====
header
======
sub-header
----------
sub-sub-header
~~~~~~~~~~~~~~
sub-sub-sub-header
^^^^^^^^^^^^^^^^^^
sub-sub-sub-sub-header
++++++++++++++++++++++
sub-sub-sub-sub-sub-header
**************************
Apache 2.0.
In alphabetical order (first name, last name, email).
- Artur Barseghyan <artur.barseghyan@gmail.com>
- Erick Martijn Bouma <bouma@gw20e.com>
- Haike Zegwaard <zegwaard@gw20e.com>
- Jasper Krebbers <krebbers@gw20e.com>
- Lukas Chripko <chripko@gw20e.com>
- Thomas Derksen <derksen@gw20e.com>
- Hans van den Berg <wepouaout64@gmail.com>