Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker and docker-compose support #13

Closed
wants to merge 12 commits into from

Conversation

nodesocket
Copy link
Contributor

@nodesocket nodesocket commented Feb 25, 2021

Docker and docker-compose support.

To build:

docker-compose build

To get a bash terminal:

docker-compose run gamestonkterminal bash

To run the app:

docker-compose up

Currently breaking 😿 🔨 with:

Creating network "gamestonkterminal_default" with the default driver
Creating gamestonkterminal_gamestonkterminal_1 ... done
Attaching to gamestonkterminal_gamestonkterminal_1
gamestonkterminal_1  | 2021-02-25 05:52:40.702552: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
gamestonkterminal_1  | 2021-02-25 05:52:40.702801: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
gamestonkterminal_1  | /home/python/.local/lib/python3.6/site-packages/fuzzywuzzy/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
gamestonkterminal_1  |   warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
gamestonkterminal_1  | /home/python/.local/lib/python3.6/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
gamestonkterminal_1  |   return torch._C._cuda_getDeviceCount() > 0
gamestonkterminal_1  | Traceback (most recent call last):
gamestonkterminal_1  |   File "gamestonk_terminal.py", line 12, in <module>
gamestonkterminal_1  |     from prediction_techniques import pred_menu as pm
gamestonkterminal_1  |   File "/home/python/prediction_techniques/pred_menu.py", line 16, in <module>
gamestonkterminal_1  |     from prediction_techniques import fbprophet
gamestonkterminal_1  |   File "/home/python/prediction_techniques/fbprophet.py", line 11, in <module>
gamestonkterminal_1  |     from fbprophet import Prophet
gamestonkterminal_1  | ModuleNotFoundError: No module named 'fbprophet'
gamestonkterminal_gamestonkterminal_1 exited with code 1

docker-compose.yaml Outdated Show resolved Hide resolved
@nodesocket nodesocket changed the title Start of Docker and docker-compose support Docker and docker-compose support Feb 25, 2021
@amingilani
Copy link

amingilani commented Feb 25, 2021

I'm getting an error on build. I wonder if you're seeing this too? 🙂 I'll investigate on my side.

➜  GamestonkTerminal git:(nodesocket) docker-compose build
Building gamestonkterminal
Step 1/9 : FROM python:3.6.13-slim-buster
 ---> 1fbc6fae5833
Step 2/9 : RUN apt-get update && apt-get -y install --no-install-recommends ca-certificates wget curl telnet net-tools iputils-ping dnsutils git gnupg unzip bzip2 && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
 ---> Using cache
 ---> e982d86d82c4
Step 3/9 : RUN useradd --create-home --shell /bin/bash python
 ---> Using cache
 ---> 0778ce38b4a6
Step 4/9 : USER python
 ---> Using cache
 ---> b12893c3b446
Step 5/9 : WORKDIR /home/python
 ---> Using cache
 ---> bb54435eef31
Step 6/9 : COPY . .
 ---> Using cache
 ---> b415e8331d1d
Step 7/9 : RUN INSTALL_ON_LINUX=1 pip install -r requirements.txt
 ---> Running in 29187875ac93
Defaulting to user installation because normal site-packages is not writeable
Collecting matplotlib==3.1.1
  Downloading matplotlib-3.1.1-cp36-cp36m-manylinux1_x86_64.whl (13.1 MB)
Collecting flair==0.7
  Downloading flair-0.7-py3-none-any.whl (448 kB)
Collecting finviz==1.3.4
  Downloading finviz-1.3.4.tar.gz (13 kB)
Collecting praw==7.1.4
  Downloading praw-7.1.4-py3-none-any.whl (155 kB)
Collecting pandas_ta==0.2.23b0
  Downloading pandas_ta-0.2.23b.tar.gz (75 kB)
Collecting numpy==1.19.2
  Downloading numpy-1.19.2-cp36-cp36m-manylinux2010_x86_64.whl (14.5 MB)
Collecting iso8601==0.1.14
  Downloading iso8601-0.1.14-py2.py3-none-any.whl (9.5 kB)
Collecting config==0.5.0.post0
  Downloading config-0.5.0.post0-py2.py3-none-any.whl (20 kB)
Collecting prettytable==2.0.0
  Downloading prettytable-2.0.0-py3-none-any.whl (22 kB)
Collecting pandas==1.1.5
  Downloading pandas-1.1.5-cp36-cp36m-manylinux1_x86_64.whl (9.5 MB)
Collecting yahoo_fin==0.8.4
  Downloading yahoo_fin-0.8.4-py3-none-any.whl (7.9 kB)
Collecting pmdarima==1.8.0
  Downloading pmdarima-1.8.0-cp36-cp36m-manylinux1_x86_64.whl (1.5 MB)
Collecting holidays==0.10.5.2
  Downloading holidays-0.10.5.2.tar.gz (121 kB)
Collecting fuzzywuzzy==0.18.0
  Downloading fuzzywuzzy-0.18.0-py2.py3-none-any.whl (18 kB)
Collecting psaw==0.0.12
  Downloading psaw-0.0.12-py3-none-any.whl (15 kB)
Collecting alpha_vantage==2.3.1
  Downloading alpha_vantage-2.3.1-py3-none-any.whl (31 kB)
Collecting pytrends==4.7.3
  Downloading pytrends-4.7.3-py3-none-any.whl (14 kB)
Collecting requests>=2.7.0
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting Quandl==3.6.0
  Downloading Quandl-3.6.0-py2.py3-none-any.whl (26 kB)
Collecting pytz==2019.3
  Downloading pytz-2019.3-py2.py3-none-any.whl (509 kB)
Collecting xlwings==0.16.4
  Downloading xlwings-0.16.4.tar.gz (635 kB)
Collecting statsmodels==0.12.2
  Downloading statsmodels-0.12.2-cp36-cp36m-manylinux1_x86_64.whl (9.5 MB)
Collecting XlsxWriter==1.2.7
  Downloading XlsxWriter-1.2.7-py2.py3-none-any.whl (141 kB)
Collecting yfinance==0.1.55
  Downloading yfinance-0.1.55.tar.gz (23 kB)
Collecting beautifulsoup4==4.9.3
  Downloading beautifulsoup4-4.9.3-py3-none-any.whl (115 kB)
Collecting FundamentalAnalysis==0.2.6
  Downloading FundamentalAnalysis-0.2.6.tar.gz (7.3 kB)
Collecting python_dateutil==2.8.1
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting scikit_learn==0.24.1
  Downloading scikit_learn-0.24.1-cp36-cp36m-manylinux2010_x86_64.whl (22.2 MB)
Collecting tensorflow==2.4.1
  Downloading tensorflow-2.4.1-cp36-cp36m-manylinux2010_x86_64.whl (394.3 MB)
Collecting aiohttp
  Downloading aiohttp-3.7.3-cp36-cp36m-manylinux2014_x86_64.whl (1.3 MB)
Collecting soupsieve>1.2
  Downloading soupsieve-2.2-py3-none-any.whl (33 kB)
Collecting lxml
  Downloading lxml-4.6.2-cp36-cp36m-manylinux1_x86_64.whl (5.5 MB)
Collecting urllib3
  Downloading urllib3-1.26.3-py2.py3-none-any.whl (137 kB)
Collecting cssselect
  Downloading cssselect-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting user_agent
  Downloading user_agent-0.1.9.tar.gz (20 kB)
Collecting torch>=1.1.0
  Downloading torch-1.7.1-cp36-cp36m-manylinux1_x86_64.whl (776.8 MB)
Killed
ERROR: Service 'gamestonkterminal' failed to build : The command '/bin/sh -c INSTALL_ON_LINUX=1 pip install -r requirements.txt' returned a non-zero code: 137

@auxon
Copy link

auxon commented Feb 25, 2021

Looking

@nodesocket
Copy link
Contributor Author

nodesocket commented Feb 25, 2021

I'm getting an error on build. I wonder if you're seeing this too? 🙂 I'll investigate on my side.

➜  GamestonkTerminal git:(nodesocket) docker-compose build
Building gamestonkterminal
Step 1/9 : FROM python:3.6.13-slim-buster
 ---> 1fbc6fae5833
Step 2/9 : RUN apt-get update && apt-get -y install --no-install-recommends ca-certificates wget curl telnet net-tools iputils-ping dnsutils git gnupg unzip bzip2 && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
 ---> Using cache
 ---> e982d86d82c4
Step 3/9 : RUN useradd --create-home --shell /bin/bash python
 ---> Using cache
 ---> 0778ce38b4a6
Step 4/9 : USER python
 ---> Using cache
 ---> b12893c3b446
Step 5/9 : WORKDIR /home/python
 ---> Using cache
 ---> bb54435eef31
Step 6/9 : COPY . .
 ---> Using cache
 ---> b415e8331d1d
Step 7/9 : RUN INSTALL_ON_LINUX=1 pip install -r requirements.txt
 ---> Running in 29187875ac93
Defaulting to user installation because normal site-packages is not writeable
Collecting matplotlib==3.1.1
  Downloading matplotlib-3.1.1-cp36-cp36m-manylinux1_x86_64.whl (13.1 MB)
Collecting flair==0.7
  Downloading flair-0.7-py3-none-any.whl (448 kB)
Collecting finviz==1.3.4
  Downloading finviz-1.3.4.tar.gz (13 kB)
Collecting praw==7.1.4
  Downloading praw-7.1.4-py3-none-any.whl (155 kB)
Collecting pandas_ta==0.2.23b0
  Downloading pandas_ta-0.2.23b.tar.gz (75 kB)
Collecting numpy==1.19.2
  Downloading numpy-1.19.2-cp36-cp36m-manylinux2010_x86_64.whl (14.5 MB)
Collecting iso8601==0.1.14
  Downloading iso8601-0.1.14-py2.py3-none-any.whl (9.5 kB)
Collecting config==0.5.0.post0
  Downloading config-0.5.0.post0-py2.py3-none-any.whl (20 kB)
Collecting prettytable==2.0.0
  Downloading prettytable-2.0.0-py3-none-any.whl (22 kB)
Collecting pandas==1.1.5
  Downloading pandas-1.1.5-cp36-cp36m-manylinux1_x86_64.whl (9.5 MB)
Collecting yahoo_fin==0.8.4
  Downloading yahoo_fin-0.8.4-py3-none-any.whl (7.9 kB)
Collecting pmdarima==1.8.0
  Downloading pmdarima-1.8.0-cp36-cp36m-manylinux1_x86_64.whl (1.5 MB)
Collecting holidays==0.10.5.2
  Downloading holidays-0.10.5.2.tar.gz (121 kB)
Collecting fuzzywuzzy==0.18.0
  Downloading fuzzywuzzy-0.18.0-py2.py3-none-any.whl (18 kB)
Collecting psaw==0.0.12
  Downloading psaw-0.0.12-py3-none-any.whl (15 kB)
Collecting alpha_vantage==2.3.1
  Downloading alpha_vantage-2.3.1-py3-none-any.whl (31 kB)
Collecting pytrends==4.7.3
  Downloading pytrends-4.7.3-py3-none-any.whl (14 kB)
Collecting requests>=2.7.0
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting Quandl==3.6.0
  Downloading Quandl-3.6.0-py2.py3-none-any.whl (26 kB)
Collecting pytz==2019.3
  Downloading pytz-2019.3-py2.py3-none-any.whl (509 kB)
Collecting xlwings==0.16.4
  Downloading xlwings-0.16.4.tar.gz (635 kB)
Collecting statsmodels==0.12.2
  Downloading statsmodels-0.12.2-cp36-cp36m-manylinux1_x86_64.whl (9.5 MB)
Collecting XlsxWriter==1.2.7
  Downloading XlsxWriter-1.2.7-py2.py3-none-any.whl (141 kB)
Collecting yfinance==0.1.55
  Downloading yfinance-0.1.55.tar.gz (23 kB)
Collecting beautifulsoup4==4.9.3
  Downloading beautifulsoup4-4.9.3-py3-none-any.whl (115 kB)
Collecting FundamentalAnalysis==0.2.6
  Downloading FundamentalAnalysis-0.2.6.tar.gz (7.3 kB)
Collecting python_dateutil==2.8.1
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting scikit_learn==0.24.1
  Downloading scikit_learn-0.24.1-cp36-cp36m-manylinux2010_x86_64.whl (22.2 MB)
Collecting tensorflow==2.4.1
  Downloading tensorflow-2.4.1-cp36-cp36m-manylinux2010_x86_64.whl (394.3 MB)
Collecting aiohttp
  Downloading aiohttp-3.7.3-cp36-cp36m-manylinux2014_x86_64.whl (1.3 MB)
Collecting soupsieve>1.2
  Downloading soupsieve-2.2-py3-none-any.whl (33 kB)
Collecting lxml
  Downloading lxml-4.6.2-cp36-cp36m-manylinux1_x86_64.whl (5.5 MB)
Collecting urllib3
  Downloading urllib3-1.26.3-py2.py3-none-any.whl (137 kB)
Collecting cssselect
  Downloading cssselect-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting user_agent
  Downloading user_agent-0.1.9.tar.gz (20 kB)
Collecting torch>=1.1.0
  Downloading torch-1.7.1-cp36-cp36m-manylinux1_x86_64.whl (776.8 MB)
Killed
ERROR: Service 'gamestonkterminal' failed to build : The command '/bin/sh -c INSTALL_ON_LINUX=1 pip install -r requirements.txt' returned a non-zero code: 137

Pull and retry. It is building the container now, but breaking when running with error ModuleNotFoundError: No module named 'fbprophet'.

@nodesocket
Copy link
Contributor Author

nodesocket commented Feb 25, 2021

@DidierRLopes I just merged the "main" branch into this branch and now getting:

ERROR: Could not find a version that satisfies the requirement scipy==1.6.1
ERROR: No matching distribution found for scipy==1.6.1

when trying to build the Docker container.

@amingilani
Copy link

Related: I added the ability to build the container with GitHub Actions.

@DidierRLopes
Copy link
Collaborator

@DidierRLopes I just merged the "main" branch into this branch and now getting:

ERROR: Could not find a version that satisfies the requirement scipy==1.6.1
ERROR: No matching distribution found for scipy==1.6.1

when trying to build the Docker container.

I've updated the requirements based on a PR, where it imports all dependencies needed. Since I've received several messages complaining about the requirements.txt.

However, in my machine, I can confirm that the scipy I have installed is 1.4.1. Can you try to build it with scipy==1.4.1. please?

I've tried to build the docker container myself and I'm getting the following error:

Killed
ERROR: Service 'gamestonkterminal' failed to build : The command '/bin/sh -c INSTALL_ON_LINUX=1 pip install -r requirements.txt' returned a non-zero code: 137

@nodesocket
Copy link
Contributor Author

@DidierRLopes changing to version 1.4.1 for scipy worked. I updated the requirements.txt to use 1.4.1.

Looks like that PR that added Levenshtein and other packages to the requirements.txt is what broke. No idea what this means:

mp.linux-x86_64-3.6/Levenshtein/_levenshtein.o
    In file included from /usr/lib/gcc/x86_64-linux-gnu/8/include-fixed/syslimits.h:7,
                     from /usr/lib/gcc/x86_64-linux-gnu/8/include-fixed/limits.h:34,
                     from /usr/local/include/python3.6m/Python.h:11,
                     from Levenshtein/_levenshtein.c:99:
    /usr/lib/gcc/x86_64-linux-gnu/8/include-fixed/limits.h:194:15: fatal error: limits.h: No such file or directory
     #include_next <limits.h>  /* recurse down to the real one */
                   ^~~~~~~~~~
    compilation terminated.
    error: command 'gcc' failed with exit status 1

Pull my branch and retry docker-compose build to see.

@DidierRLopes
Copy link
Collaborator

Thank you, this is much appreciated!

Let me give it a go.

Dockerfile Outdated Show resolved Hide resolved
@nodesocket
Copy link
Contributor Author

Ok, we getting closer with my latest commit. I think Carbon is some dependency needed.

  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.6m -c appscript_3x/ext/ae.c -o build/temp.linux-x86_64-3.6/appscript_3x/ext/ae.o -DMAC_OS_X_VERSION_MIN_REQUIRED=MAC_OS_X_VERSION_10_6 -D__LP64__
  In file included from appscript_3x/ext/ae.c:16:
  appscript_3x/ext/ae.h:26:10: fatal error: Carbon/Carbon.h: No such file or directory
   #include <Carbon/Carbon.h>
            ^~~~~~~~~~~~~~~~~
  compilation terminated.
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for appscript

@DidierRLopes
Copy link
Collaborator

The docker is still failing on my end, but at the build.

I'm not familiar with docker images. so I need to try to understand what's going on better, and then make sure everything still works smoothly.

I'm starting work soon, and will be again available at the end of the day to continue having a look at this.

Again, I appreciate a lot your help with this! Thanks.

@micheloosterhof
Copy link

I got it running in Docker, by removing the appscript dependency. This module is only supported on MacOS anyway.
I went to scipy==1.5.4.

I also updated pip, setuptools and wheel, and ran the entire dependencies in a virtual environment:

RUN cd /home/python && \
    python3 -m venv gst-env && \
    . gst-env/bin/activate && \
    pip install --no-cache-dir --upgrade pip && \
    pip install --no-cache-dir --upgrade setuptools wheel && \
    INSTALL_ON_LINUX=1 pip install --no-cache-dir --upgrade -r requirements.txt

And it runs.

@nodesocket
Copy link
Contributor Author

@micheloosterhof why create and use a venv if this is in Docker? What’s the benefit?

I will try removing appscript and bumping scipy to version 1.5.4.

@thekaveman
Copy link

thekaveman commented Feb 25, 2021

Just a heads up, scipy 1.6.1 requires Python >= 3.7.

I was looking at Dockerizing before I saw this PR, let me know if there's any help I can provide!

@micheloosterhof
Copy link

@micheloosterhof why create and use a venv if this is in Docker? What’s the benefit?

More control over your Python environment, no interference with the system provided Python libraries.
This also keeps the same workflow for both normal installs (without Docker) and Docker container for easier maintenance. If you google virtualenv and docker you'll find a few more reasons. Ultimately I like to do it for consistency.

@derekslenk
Copy link

I got it running in Docker, by removing the appscript dependency. This module is only supported on MacOS anyway.
I went to scipy==1.5.4.

I also updated pip, setuptools and wheel, and ran the entire dependencies in a virtual environment:

RUN cd /home/python && \
    python3 -m venv gst-env && \
    . gst-env/bin/activate && \
    pip install --no-cache-dir --upgrade pip && \
    pip install --no-cache-dir --upgrade setuptools wheel && \
    INSTALL_ON_LINUX=1 pip install --no-cache-dir --upgrade -r requirements.txt

And it runs.

Hmm, I ended up having to go with

    cd gst && \
    python3 -m venv gst-env && \
    . gst-env/bin/activate && \
    pip install --no-cache-dir --upgrade pip && \
    pip install --no-cache-dir numpy==1.19.2 && \
    pip install --no-cache-dir --upgrade setuptools wheel matplotlib fbprophet && \
    INSTALL_ON_LINUX=1 pip install --no-cache-dir --upgrade -r requirements.txt

But I was using ubuntu as a base as I am a noob

@shadycuz
Copy link
Contributor

@nodesocket Thanks for your work getting docker up and running. We had to made some tiny changes because we are moving the project to poetry. You can see the changes here

If you change your PR so that collaborators can edit your branch, we can merge our changes here in this PR.

The only concern I have is around charts. The charts in GamestonkTerminal require a display so that matplot lib can draw. With docker that doesn't happen. The commands seem to work fine but the charts aren't shown. It is possible to attach a display to a docker container see but it seems really complicated for most users and kinda against the point of leveraging docker in the first place.

With the change to poetry we are working hard on making GamestonkTerminal installable on multiple versions of python. I think in the future we could change the code so that it either doesn't draw charts if no display is detected or maybe draws ASCII charts or something? Then docker becomes a much more viable option.

Really interested in your thoughts on this and really appreciate the effort you put into this PR.

🚀 🚀 GamestonkTerminal to the 🌔

@nodesocket
Copy link
Contributor Author

@shadycuz how do I allow collaborators to edit this branch? Sorry, not seeing that option.

@nodesocket
Copy link
Contributor Author

@shadycuz I think it is already enabled?

Screen Shot 2021-02-28 at 6 18 38 PM

@DidierRLopes
Copy link
Collaborator

Hey @nodesocket @amingilani @auxon @micheloosterhof ,

Thank you for this! Me and @shadycuz added on top of this (https://github.com/DidierRLopes/GamestonkTerminal/pull/76) and have now pushed the changes to the main branch.

Let me know if you have any issue.

@nodesocket nodesocket deleted the docker branch March 2, 2021 23:21
DidierRLopes pushed a commit that referenced this pull request Apr 19, 2021
DidierRLopes pushed a commit that referenced this pull request Jan 12, 2022
piiq pushed a commit that referenced this pull request Nov 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

9 participants