Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: use separate docker image for mlflow server #101

Merged
merged 9 commits into from
Nov 29, 2022
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 1 addition & 2 deletions .flake8
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
[flake8]
ignore = |
E203, F821
ignore = E203, F821
exclude = .git,__pycache__,docs/source/conf.py,old,build,dist
max-complexity = 10
max-line-length = 100
8 changes: 3 additions & 5 deletions docs/quick-start.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,10 +72,8 @@ Once Numaflow is installed, create a simple Numalogic pipeline, which takes in t

For building this pipeline, navigate to [numalogic-simple-pipeline](https://github.com/numaproj/numalogic/tree/main/examples/numalogic-simple-pipeline) under the examples folder and execute the following commands.

1. Build the docker image, import it to k3d, and apply the pipeline. *Note Make sure the pipeline and, numaflow controllers and isbsvc pods are running in the same namespace (`default` in this case).*
1. Apply the pipeline. *Note Make sure the pipeline and, numaflow controllers and isbsvc pods are running in the same namespace (`default` in this case).*
```shell
docker build -t numalogic-simple-pipeline:v1 . && k3d image import docker.io/library/numalogic-simple-pipeline:v1

kubectl apply -f numa-pl.yaml
```
2. To verify if the pipeline has been deployed successfully, check the status of each pod.
Expand All @@ -101,9 +99,9 @@ numalogic-simple-pipeline-in-0-tmd0v 1/1 Running 0

Once the pipeline has been created, the data can be sent to the pipeline by port-forwarding the input vertex.

1. Port-forward to the http-source vertex
1. Port-forward to the http-source vertex. From the above pod output, this would be:
```shell
kubectl port-forward numalogic-simple-pipeline-in-0-xxxxx 8443
kubectl port-forward numalogic-simple-pipeline-in-0-tmd0v 8443
```

2. Send the data to the pod via curl
Expand Down
6 changes: 6 additions & 0 deletions examples/numalogic-simple-pipeline/.dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
.git/
__pycache__/
**/__pycache__/
*.py[cod]
*$py.class
.idea/
31 changes: 25 additions & 6 deletions examples/numalogic-simple-pipeline/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
FROM python:3.9.12-slim
FROM python:3.10-slim-bullseye AS builder

ENV PYTHONFAULTHANDLER=1 \
PYTHONUNBUFFERED=1 \
PYTHONHASHSEED=random \
PIP_NO_CACHE_DIR=off \
PIP_NO_CACHE_DIR=on \
PIP_DISABLE_PIP_VERSION_CHECK=on \
PIP_DEFAULT_TIMEOUT=100 \
POETRY_VERSION=1.2.2 \
Expand All @@ -13,6 +14,7 @@ ENV PYTHONFAULTHANDLER=1 \
VENV_PATH="/opt/pysetup/.venv"

ENV PATH="$POETRY_HOME/bin:$VENV_PATH/bin:$PATH"

RUN apt-get update \
&& apt-get install --no-install-recommends -y \
curl \
Expand All @@ -25,14 +27,31 @@ RUN apt-get update \
# install dumb-init
&& wget -O /dumb-init https://github.com/Yelp/dumb-init/releases/download/v1.2.5/dumb-init_1.2.5_x86_64 \
&& chmod +x /dumb-init \
\
# install poetry - respects $POETRY_VERSION & $POETRY_HOME
&& curl -sSL https://install.python-poetry.org | python3 -

FROM builder AS mlflow

WORKDIR $PYSETUP_PATH
COPY ./pyproject.toml ./poetry.lock ./
RUN poetry install --only mlflowserver --no-cache --no-root && \
rm -rf ~/.cache/pypoetry/

ADD . /app
WORKDIR /app

RUN chmod +x entry.sh

ENTRYPOINT ["/dumb-init", "--"]
CMD ["/app/entry.sh"]

EXPOSE 5000


FROM builder AS udf

WORKDIR $PYSETUP_PATH
COPY ./pyproject.toml ./poetry.lock $PYSETUP_PATH
RUN poetry install --without dev --no-cache --no-root && \
COPY ./pyproject.toml ./poetry.lock ./
RUN poetry install --without mlflowserver --no-cache --no-root && \
rm -rf ~/.cache/pypoetry/

ADD . /app
Expand Down
10 changes: 5 additions & 5 deletions examples/numalogic-simple-pipeline/numa-pl.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ spec:
min: 1
udf:
container:
image: docker.io/library/numalogic-simple-pipeline:v1
image: quay.io/numaio/numalogic/example-udf
env:
- name: WIN_SIZE
value: "12"
Expand All @@ -31,7 +31,7 @@ spec:
min: 1
udf:
container:
image: docker.io/library/numalogic-simple-pipeline:v1
image: quay.io/numaio/numalogic/example-udf
env:
- name: WIN_SIZE
value: "12"
Expand All @@ -44,7 +44,7 @@ spec:
min: 1
udf:
container:
image: docker.io/library/numalogic-simple-pipeline:v1
image: quay.io/numaio/numalogic/example-udf
env:
- name: WIN_SIZE
value: "12"
Expand All @@ -65,7 +65,7 @@ spec:
env:
- name: WIN_SIZE
value: "12"
image: docker.io/library/numalogic-simple-pipeline:v1
image: quay.io/numaio/numalogic/example-udf
args:
- python
- starter.py
Expand Down Expand Up @@ -106,7 +106,7 @@ spec:
app: mlflow
spec:
containers:
- image: docker.io/library/numalogic-simple-pipeline:v1
- image: quay.io/numaio/numalogic/example-mlflow
name: mlflow
args:
- server
Expand Down