Skip to content

Latest commit

 

History

History
184 lines (161 loc) · 9.83 KB

README.md

File metadata and controls

184 lines (161 loc) · 9.83 KB
Raw

hyadata

ci license docker react hyadata

Useage

Option 1:

Using a docker.
First, you should install docker.

Now, for exploring with GUI webapp, the steps are:

# pulling the images  
docker pull shaharjacob/msbackend  
docker pull shaharjacob/mswebapp  

# crating a network to combine both containers  
docker network create "FAME"  

# running both containers
docker run -d -p 5031:5031 --name backend --network "FAME" -e "FLASK_ENV=development" -e "FLASK_APP=app.app" -e "SENTENCE_TRANSFORMERS_HOME=cache" shaharjacob/msbackend flask run --host "0.0.0.0" --port 5031  
docker run -d -p 3000:3000 --name webapp --network "FAME" shaharjacob/mswebapp npm start

After few seconds you can open browser at: http://localhost:3000/
The first run may take a few second because it download the sBERT model into the container.

Verbose:
If you want to make sure that webapp container load successfully:

docker logs -f webapp

If you want to see backend verbose, use:

docker logs -f backend

Exit from the verbose mode can be done with ctrl + c.
 

If you want to run without the demo, you need to connect to the backend container.
To do that you just need to type:

docker exec -it backend bash

You should be now in backend folder, and should run:

python evaluation/evaluation.py --yaml playground.yaml

More details about the execute command can be found under Execute section.
 

Option 2:

Install dependencies and run on your local PC.

git clone https://github.com/shaharjacob/FAME.git
cd FAME
pip install -r requirements.txt

Now you ready for the execute command detailed in Execute section.
 

FOR GUI ONLY
First option, working with docker. Just run:

docker-compose -f docker-compose.yml up -d

Then open the browser at: http://localhost:3000
 

Alternatively (without docker at all), you can do the following steps:

  1. Install Node.js, make sure its in your PATH. Install version 16.13.0.
  2. Now we need to install the react dependencies:
cd webapp
npm ci
  1. In pakeage.json, change the proxy from http://backend:5031 to http://localhost:5031, the 'backend' is necessary when running the docker.
  2. Now back to the root folder, and open the file ./backend/app/app.py, and uncomment the if main == ... section below.
  3. from the root folder, run:
python backend/app/app.py
  1. Now we just need to start the frontend:
cd webapp
npm start

 

Option 3:

Running on the university cluster using exsiting repo (without demo, only backend).

  1. ssh to phoenix cluster.
  2. cd /cs/labs/dshahaf/shahar.jacob/FAME
  3. Edit the shell script under the root folder called runme.sh with the command you want to run (see Execute section).
  4. Run the following command:
sbatch --mem=6gm -c2 --time=12:0:0 --gres=gpu:2 --verbose "runme.sh"

See the log with the command:

tail -f slurm-{job-id}.out

The job id shown when the job in submitted.
 

Execute

Execution is done by configure a yaml file. Examples for yaml files can be found under: backend/evalution, in particular you can use backend/evalution/playground.yaml.
You can see inside this file a template (in comment), and another example.
After editting the yaml by adding another entry, you can use the following command:

python backend/evaluation/evaluation.py --yaml playground.yaml

If you want the suggestions to be available, add --suggestions. This is not recommend unless you looking for suggstions.
By default, the script is running all the entries in the yaml. If you want to run specific entry, use --specify {entry number, start from 1}. You can specify muliple entries.
For example, running the first entry only:

python backend/evaluation/evaluation.py --yaml playground.yaml --specify 1

Running the first and the third entries:

python backend/evaluation/evaluation.py --yaml playground.yaml --specify 1 --specify 3

 

Troubleshooting

M1 apple

For transformers: bash``` curl https://sh.rustup.rs -sSf | bash -s

Restart the terminal, then install again.

For sklearn:
```bash
pip install --no-cache --no-use-pep517 pythran cython pybind11 gast"==0.4.0"
pip install --pre -i https://pypi.anaconda.org/scipy-wheels-nightly/simple scipy
pip install --no-use-pep517 scikit-learn"==1.0.0"

For sentence_transformers:

# first need to install brew
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" 2> /dev/null
# after that you should add brew to your PATH
# now install sentencepiece
brew install sentencepiece

# now we can install sentence_transformers
pip install sentence_transformers

docker

in your Dockerfile, add the following lines before the pip installations:

RUN apt-get update
RUN apt-get install -y \
    build-essential \
    curl
RUN apt-get update

# Get Rust
RUN curl https://sh.rustup.rs -sSf | bash -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"

Then, you should build again the backend:

docker-compose build backend