Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Structure] Dockerizing the project #30

Merged
merged 35 commits into from
Aug 14, 2021
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
e72c56f
Removed root requirements, split amongst containers
Rubix982 Jul 23, 2021
fb8db43
Changed backend directory structure
Rubix982 Jul 23, 2021
a5fdc32
Shifted location of data to backend/data/
Rubix982 Jul 23, 2021
4f68646
Changed port for backend
Rubix982 Jul 23, 2021
646e4d8
Add Dockerfiles for containerziation, Dock Compose for container orch…
Rubix982 Jul 23, 2021
91a51cb
Introduce policy with deal with Docker Containers
Rubix982 Jul 23, 2021
0dc058e
Bug fix, random chown data line
Rubix982 Jul 23, 2021
b7bc9c1
Exposing PORT
Rubix982 Jul 23, 2021
c3d8e42
Tag "image" for the containers
Rubix982 Jul 23, 2021
67e76f8
Added FastAPI as dependency, required by Jina
Rubix982 Jul 23, 2021
1180c08
Added aiohttp dependency, remove app.py for testing
Rubix982 Jul 23, 2021
5b6f162
Delete pdb_data_seq.csv
georgeamccarthy Jul 23, 2021
b684f13
Added .csv and embeddings to .gitignore
Rubix982 Jul 23, 2021
090d34b
Cache testing done for backend/Dockfile
Rubix982 Jul 23, 2021
a385964
Merge branch 'Dockerizing' of github.com:Rubix982/protein_search into…
Rubix982 Jul 23, 2021
739744d
Removed data entity from backend, added to .gitignore
Rubix982 Aug 2, 2021
2092242
Deal with downloading data/ and creating pdf_data_seq.csv
Rubix982 Aug 4, 2021
50c377a
Added directories to ignore, more instructions to "docker" policy, do…
Rubix982 Aug 6, 2021
0d5dc23
Refactored frontend, change docker base image
Rubix982 Aug 6, 2021
4536403
Formatted, linted "tests" directory
Rubix982 Aug 6, 2021
69d7f5b
Finalized Dockerfile with model and tokenizer caching
Rubix982 Aug 6, 2021
c065ca4
Refactored, logged, cache logic implemented, added util
Rubix982 Aug 6, 2021
f2996cb
Lint & format policy modified
Rubix982 Aug 6, 2021
1a4c816
Introduced instructions or working with Docker
Rubix982 Aug 6, 2021
9fef8d4
Fixed connection bug for the frontend to the backend
Rubix982 Aug 6, 2021
84e8d41
Renamed "my_executors.py" to "executors.py", fixed imports
Rubix982 Aug 6, 2021
d545e80
Fetching & Caching prior to Flow start implemented
Rubix982 Aug 6, 2021
2ee1591
Update README.md
Rubix982 Aug 6, 2021
f516173
Dockerizing and restructuring.
fissoreg Aug 6, 2021
c1d9c6f
Merged conflicts.
fissoreg Aug 9, 2021
3c4cb8c
Fix tests + search test + formatting.
fissoreg Aug 10, 2021
22519cb
1. Shuffling proteins.
fissoreg Aug 11, 2021
a061368
1. Got rid of static method to load ProtBERT model.
fissoreg Aug 12, 2021
ac8d9c2
Fixed connection between frontend and backend.
fissoreg Aug 13, 2021
c292330
Merge pull request #2 from fissoreg/Rubix982-Dockerizing
Rubix982 Aug 13, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
16 changes: 16 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,19 @@ test: ## Run tests
build:
make format
make coverage

# For building the docker compose
docker:
docker-compose -f docker-compose.yml up --build

# For starting the docker comppose,
up:
docker-compose -f docker-compose.yml up

## For removing containers
remove:
docker-compose down --remove-orphans

# List all containers
ps:
docker-compose ps -a
33 changes: 33 additions & 0 deletions backend/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Get image for Python
FROM python:3.8-slim-buster

# Set working directory
WORKDIR /app/

# pip does not like being run as root, so create a user
RUN useradd --create-home jina

# Add the data folder locally to container
COPY . .

# Give read permissions for the data folder
RUN chown jina data

# Switch to user
USER jina

# Path change needed for huggingface-cli and jina
ENV PYTHONPATH "${PYTHONPATH}:/home/jina/.local/bin"
ENV PATH "${PATH}:/home/jina/.local/bin"
Rubix982 marked this conversation as resolved.
Show resolved Hide resolved

# Add requirement contents
COPY ./requirements.txt requirements.txt

# Install dependencies
RUN pip3 install -r requirements.txt

# Expose port
EXPOSE 8020

# Run the application
CMD [ "python", "src/app.py" ]
File renamed without changes.
File renamed without changes.
File renamed without changes.
20 changes: 20 additions & 0 deletions backend/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
torch==1.9.0
transformers==4.9.0
jina==2.0.10
asgiref==3.4.1
click==8.0.1
h11==0.12.0
uvicorn==0.14.0
fastapi==0.67.0
pydantic==1.8.2
starlette==0.14.2
typing-extensions==3.10.0.0
uvloop==0.15.3
aiohttp==3.7.4.post0
async-timeout==3.0.1
attrs==21.2.0
chardet==4.0.0
idna==3.2
multidict==5.1.0
typing-extensions==3.10.0.0
yarl==1.6.3
2 changes: 1 addition & 1 deletion backend/app.py → backend/src/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ def main():
proteins = DocumentArray(docs_generator)[0:42]

flow = (
Flow(port_expose=12345, protocol='http')
Flow(port_expose=8020, protocol='http')
.add(uses=ProtBertExecutor)
.add(uses=MyIndexer)
)
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
31 changes: 31 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
version: "3"

services:

backend:
image: rubix982/jina-backend:v1.0.0
fissoreg marked this conversation as resolved.
Show resolved Hide resolved
container_name: protein-search-backend
build:
context: ./backend
stdin_open: true
restart: always
volumes:
- "./backend:/app"
- "./backend/requirements.txt:/app/requirements.txt"
ports:
- "8020:8020"

frontend:
image: rubix982/jina-frontend:v1.0.0
container_name: protein-search-frontend
build:
context: ./frontend
stdin_open: true
restart: always
volumes:
- "./frontend:/app"
- "./frontend/requirements.txt:/app/requirements.txt"
ports:
- "8501:8501"
depends_on:
- "backend"
30 changes: 30 additions & 0 deletions frontend/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Get image for Python
FROM python:3.8-slim-buster

# Set working directory
WORKDIR /app/

# pip does not like being run as root, so create a user
RUN useradd --create-home jina

# Add the data folder locally to container
COPY . .

# Switch to user
USER jina

# Path change needed for huggingface-cli and jina
ENV PYTHONPATH "${PYTHONPATH}:/home/jina/.local/bin"
ENV PATH "${PATH}:/home/jina/.local/bin"

# Add requirement contents
COPY ./requirements.txt requirements.txt

# Install dependencies
RUN pip3 install -r requirements.txt

# Expose Port
EXPOSE 8501

# Run the application
CMD [ "streamlit", "run", "app.py" ]
2 changes: 1 addition & 1 deletion frontend/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ def protein_3d(pdb_id="1YCR", width=200, height=200):
width=width,
)

endpoint = "http://localhost:12345/search"
endpoint = "http://localhost:8020/search"

st.title("Proteins Search")

Expand Down
2 changes: 2 additions & 0 deletions frontend/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
streamlit==0.85.0
requests==2.26.0
4 changes: 0 additions & 4 deletions requirements.txt

This file was deleted.