-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
add a script to run docker and mount the current directories (#1)
- Loading branch information
Showing
2 changed files
with
45 additions
and
17 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,50 +1,61 @@ | ||
#Introduction# | ||
Usagi | ||
=== | ||
|
||
Usagi is a data discovery system for Recruit’s internal infrastructure. Usagi crawls metadata within Recruit's web services every day and builds a catalog of data sets. Usagi enables users to search, monitor, and annotate the metadata which helps them discover the appropriate data to perform analysis. | ||
|
||
Many large enterprises today witness an explosion in the number of data sets. Therefore, we aim to make Usagi open source. We hope Usagi helps data-users in these enterprises discover meaningful data easily. The members who invent Usagi are contributors of Meta-Looking in Recruit Holdings Co., Ltd. and GOODS [1]. | ||
|
||
[1] Halevy, Alon, et al. "Goods: Organizing Google's Datasets." Proceedings of the 2016 International Conference on Management of Data. ACM, 2016. | ||
|
||
#Setup Usagi# | ||
A. Write a configuration file to teach Usagi to access the databases which you want to manage. (In the future, Usagi plans to also take metadata directly to avoid accessing user databases if needed for security reasons.) For example, | ||
|
||
~~~ini | ||
# Setup Usagi | ||
* A. Write a configuration file to teach Usagi to access the databases which you want to manage. (In the future, Usagi plans to also take metadata directly to avoid accessing user databases if needed for security reasons.) For example, | ||
|
||
```ini | ||
[ClassicModels] | ||
data_store: psql | ||
host: localhost | ||
port: 5432 | ||
user: classic_user | ||
dbname: classicmodels | ||
password: password | ||
~~~ | ||
``` | ||
|
||
B. Execute the Usagi installer: | ||
* B. Execute the Usagi installer: | ||
|
||
~~~bash | ||
```bash | ||
./usagi-installer --data-store classic_models.cfg | ||
~~~ | ||
``` | ||
|
||
C. Start Solr server and JSON API | ||
* C. Start Solr server and JSON API | ||
|
||
~~~bash | ||
```bash | ||
./solr/solr/bin/solr start | ||
cd api | ||
pkill -KILL api_server.py | ||
python api_server.py & | ||
cd .. | ||
~~~ | ||
``` | ||
|
||
D. Try to search | ||
* D. Try to search | ||
|
||
~~~bash | ||
```bash | ||
cd importer | ||
python search.py <your_favourite_keyword> | ||
~~~ | ||
``` | ||
|
||
E. Try JSON API | ||
* E. Try JSON API | ||
|
||
~~~bash | ||
```bash | ||
curl http://localhost:8085/api/search?q=<your_favourite_keyword> | ||
~~~ | ||
``` | ||
|
||
# Run test | ||
|
||
```bash | ||
./test/run_test.sh | ||
|
||
# create and run a docker container.... | ||
|
||
pytest # in the docker container | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
#!/bin/bash | ||
set -veu | ||
|
||
ROOT_DIR=$(pwd) | ||
|
||
if [[ "$(docker images -q usagi/test:latest 2> /dev/null)" == "" ]]; then | ||
docker build -f Dockerfile-forTest -t usagi/test . | ||
fi | ||
|
||
docker run -it --rm \ | ||
-v ${ROOT_DIR}/test:/usagi/test \ | ||
-v ${ROOT_DIR}/api:/usagi/api \ | ||
-v ${ROOT_DIR}/importer:/usagi/importer \ | ||
-v ${ROOT_DIR}/installer:/usagi/installer \ | ||
-v ${ROOT_DIR}/command:/usagi/command \ | ||
-v ${ROOT_DIR}/config:/usagi/config \ | ||
usagi/test |