Reorganize project extras for a pip installable package (#3432)

* Re-solve the dependencies for optimization extras * Remove unused reportlab helpers * Update install from source and troubleshooting docs * Fix typos found by cspell * Prevent po menu from launching when optimization toolkit not installed * Add optimization dependencies to default requirements.txt for the CI * Add optimization toolkit installation to the CI * Fix markdown linting * Update CI test dependency installation for matrix tests * Fix markdown linting * Update source installation docs * Clean up ToC in the main readme * Update the SDK documentation * Fix markdown linting after making readme sections linkable * Fix more markdown linting * Fix more markdown linting * Update secrets detection with exclusion rules * Bump detect-secrets configuration * Tune secret detector * Fix more markdown linting * Add optimization toolkit install to installer GH actions * Add optimization toolkit install to docker build * Fix typos * Add "all" extras setting to install torch and riskfolio * Move forecasting dependencies to pyproject.toml * Update CI build configs * Update documentation to feature the "all" extra * Lint markdown * Update docs to specify requirement of having a C compiler * Added log Id to codesigning steps in inteel Macos workflow * Changed disk for the m1 macos build * Changed disk for the m1 macos build * Changed disk for the intel macos build * taking out sigtool removal line * Remove dev version number from pyproject.toml * removing sigtool from all build steps * changes to m1 build and commenting test_report_economy * more m1 build changes * testing exit on conda setup * some more * changing shell to bash to setup conda * bash on lines * bash on all lines * small bash removals * intel fix and test_crypto int test * upgrading setupminiconda version * adding back in intel conda init and reset * intel removal of init * taking bash back out from m1 * removing old codesign lines * adding codesign specific binaries * ssl fix for all terminal * removing if installer * various ssl fixes across commands * fixing ssl mistake * integration test changes * Linting + remove comment * Update poetry.lock * Update requirements * Update the docs to mention that PO is not working on arm arch * Update deps to pin setuptools * Disable reports on pyinstaller packaged version * Update reports * small grammar change for forecast report message * Pin grpcio following introduced regression in 1.51.0 * Update docs to reflect changes in how optimization is installed Co-authored-by: Luqman Bello <lb@Luqmans-MacBook-Pro.local> Co-authored-by: James Simmons <simmonsj330@gmail.com> Co-authored-by: James Simmons <53658028+simmonsj330@users.noreply.github.com> Co-authored-by: James Maslek <jmaslek11@gmail.com> Co-authored-by: Chavithra <chavithra@gmail.com>
OpenBB-finance · Nov 22, 2022 · 15d1870 · 15d1870
1 parent 2e7eb4c
commit 15d1870
Show file tree

Hide file tree

Showing 31 changed files with 2,168 additions and 4,547 deletions.
diff --git a/.github/workflows/intel_macos_build.yml b/.github/workflows/intel_macos_build.yml
@@ -34,7 +34,7 @@ jobs:
           key: conda-macos-3-9-${{ hashFiles('build/conda/conda-3-9-env-full.yaml') }}
       # Set up miniconda using the environment yaml file within the repo
       - name: Setup Miniconda
-        uses: conda-incubator/setup-miniconda@v2
+        uses: conda-incubator/setup-miniconda@v2.2.0
         with:
           miniconda-version: "latest"
           auto-update-conda: true
@@ -44,28 +44,19 @@ jobs:
           environment-file: build/conda/conda-3-9-env-full.yaml
           activate-environment: build_env
           use-only-tar-bz2: true # Needed for caching some reason
-      - name: Initialize Conda and Reset Terminal
-        run: |
-          conda init
-          exec zsh -l
-          source ~/.zshrc
       - name: Run Poetry
         run: poetry install
       - name: Install Pyinstaller
         run: poetry install -E installer
-      - name: Install Prediction Modules
-        run: poetry install -E prediction
+      - name: Install Portfolio Optimization and Forecasting Toolkits
+        run: poetry install -E all
       - name: Install Specific Papermill
         run: |
           pip uninstall papermill -y
           pip install git+https://github.com/nteract/papermill.git@main
-      # Double Checking Version Number
-      - name: Check OpenBB Version
-        run: pip list | grep openbb
       # This is basically the same as build4mac.sh
       - name: Build Bundle
         run: |
-          conda remove --force sigtool -y
           pyinstaller build/pyinstaller/terminal.spec --clean
           osascript build/pyinstaller/setup_icons.applescript
           mkdir DMG
@@ -77,7 +68,7 @@ jobs:
           MACOS_CERTIFICATE_PWD: ${{ secrets.MACOS_CERTIFICATE_PWD }}
           MACOS_KEYCHAIN_PWD: ${{ secrets.MACOS_KEYCHAIN_PWD }}
           MACOS_CODESIGN_IDENTITY: ${{ secrets.MACOS_CODESIGN_IDENTITY }}
-        run: | # when pushing to main, make to generate new cert, and utilize secrets to store newpassowrd, and identity
+        run: | # when pushing to main, make to generate new cert, and utilize secrets to store new password, and identity
           echo "Decoding"
           echo $MACOS_CERTIFICATE | base64 --decode > certificate.p12
           echo "Creating Keychain"
@@ -109,14 +100,19 @@ jobs:
           /usr/bin/codesign --deep --force --verify --verbose --options runtime --entitlements "build/pyinstaller/entitlements.plist" -s $MACOS_CODESIGN_IDENTITY DMG/OpenBB\ Terminal/.OpenBB/*/*.dylib
           /usr/bin/codesign --deep --force --verify --verbose --options runtime --entitlements "build/pyinstaller/entitlements.plist" -s $MACOS_CODESIGN_IDENTITY DMG/OpenBB\ Terminal/.OpenBB/*/*/*.dylib
           /usr/bin/codesign --deep --force --verify --verbose --options runtime --entitlements "build/pyinstaller/entitlements.plist" -s $MACOS_CODESIGN_IDENTITY DMG/OpenBB\ Terminal/.OpenBB/*/*/*/*/*.dylib
-          /usr/bin/codesign --deep --force --verify --verbose --options runtime --entitlements "build/pyinstaller/entitlements.plist" -s $MACOS_CODESIGN_IDENTITY DMG/OpenBB\ Terminal/.OpenBB/scipy/.dylibs/*.dylib
+          /usr/bin/codesign --deep --force --verify --verbose --options runtime --entitlements "build/pyinstaller/entitlements.plist" -s $MACOS_CODESIGN_IDENTITY DMG/OpenBB\ Terminal/.OpenBB/prophet/.dylibs/*.dylib
+          /usr/bin/codesign --deep --force --verify --verbose --options runtime --entitlements "build/pyinstaller/entitlements.plist" -s $MACOS_CODESIGN_IDENTITY DMG/OpenBB\ Terminal/.OpenBB/prophet/stan_model/cmdstan-2.26.1/stan/lib/stan_math/lib/tbb/*.dylib
           /usr/bin/codesign --deep --force --verify --verbose --options runtime --entitlements "build/pyinstaller/entitlements.plist" -s $MACOS_CODESIGN_IDENTITY DMG/OpenBB\ Terminal/.OpenBB/torch/.dylibs/*.dylib
           echo "Code Signing Other Files"
+          /usr/bin/codesign --deep --force --verify --verbose --options runtime --entitlements "build/pyinstaller/entitlements.plist" -s $MACOS_CODESIGN_IDENTITY DMG/OpenBB\ Terminal/.OpenBB/prophet/stan_model/prophet_model.bin
+          /usr/bin/codesign --deep --force --verify --verbose --options runtime --entitlements "build/pyinstaller/entitlements.plist" -s $MACOS_CODESIGN_IDENTITY DMG/OpenBB\ Terminal/.OpenBB/prophet/stan_model/cmdstan-2.26.1/bin/diagnose
+          /usr/bin/codesign --deep --force --verify --verbose --options runtime --entitlements "build/pyinstaller/entitlements.plist" -s $MACOS_CODESIGN_IDENTITY DMG/OpenBB\ Terminal/.OpenBB/prophet/stan_model/cmdstan-2.26.1/bin/print
+          /usr/bin/codesign --deep --force --verify --verbose --options runtime --entitlements "build/pyinstaller/entitlements.plist" -s $MACOS_CODESIGN_IDENTITY DMG/OpenBB\ Terminal/.OpenBB/prophet/stan_model/cmdstan-2.26.1/bin/stanc
+          /usr/bin/codesign --deep --force --verify --verbose --options runtime --entitlements "build/pyinstaller/entitlements.plist" -s $MACOS_CODESIGN_IDENTITY DMG/OpenBB\ Terminal/.OpenBB/prophet/stan_model/cmdstan-2.26.1/bin/stansummary
           /usr/bin/codesign --deep --force --verify --verbose --options runtime --entitlements "build/pyinstaller/entitlements.plist" -s $MACOS_CODESIGN_IDENTITY DMG/OpenBB\ Terminal/.OpenBB/pyarrow/plasma-store-server
           /usr/bin/codesign --deep --force --verify --verbose --options runtime --entitlements "build/pyinstaller/entitlements.plist" -s $MACOS_CODESIGN_IDENTITY DMG/OpenBB\ Terminal/.OpenBB/torch/bin/protoc
           /usr/bin/codesign --deep --force --verify --verbose --options runtime --entitlements "build/pyinstaller/entitlements.plist" -s $MACOS_CODESIGN_IDENTITY DMG/OpenBB\ Terminal/.OpenBB/torch/bin/protoc-3.13.0.0
           /usr/bin/codesign --deep --force --verify --verbose --options runtime --entitlements "build/pyinstaller/entitlements.plist" -s $MACOS_CODESIGN_IDENTITY DMG/OpenBB\ Terminal/.OpenBB/torch/bin/torch_shm_manager
-          /usr/bin/codesign --deep --force --verify --verbose --options runtime --entitlements "build/pyinstaller/entitlements.plist" -s $MACOS_CODESIGN_IDENTITY DMG/OpenBB\ Terminal/.OpenBB/prophet/stan_model/prophet_model.bin
           echo "Code Sign OpenBB Executable File"
           /usr/bin/codesign --deep --force --verify --verbose --options runtime --entitlements "build/pyinstaller/entitlements.plist" -s $MACOS_CODESIGN_IDENTITY DMG/OpenBB\ Terminal/.OpenBB/OpenBBTerminal
       - name: Create DMG

diff --git a/.github/workflows/m1_macos_build.yml b/.github/workflows/m1_macos_build.yml
@@ -34,7 +34,7 @@ jobs:
           key: conda-macos-3-9-${{ hashFiles('build/conda/conda-3-9-env-full.yaml') }}
       # Set up miniconda using the environment yaml file within the repo
       - name: Setup Miniconda
-        uses: conda-incubator/setup-miniconda@v2
+        uses: conda-incubator/setup-miniconda@v2.2.0
         with:
           miniconda-version: "latest"
           auto-update-conda: true
@@ -43,29 +43,21 @@ jobs:
           channel-priority: flexible
           environment-file: build/conda/conda-3-9-env-full.yaml
           activate-environment: build_env
+          auto-activate-base: false
           use-only-tar-bz2: true # Needed for caching some reason
-      - name: Initialize Conda and Reset Terminal
-        run: |
-          conda init
-          exec zsh -l
-          source ~/.zshrc
       - name: Run Poetry
         run: poetry install
       - name: Install Pyinstaller
         run: poetry install -E installer
-      - name: Install Prediction Modules
-        run: poetry install -E prediction
+      - name: Poetry Install Portfolio Optimization and Forecasting Toolkits
+        run: poetry install -E all
       - name: Install Specific Papermill
         run: |
           pip uninstall papermill -y
           pip install git+https://github.com/nteract/papermill.git@main
-      # Double Checking Version Number
-      - name: Check OpenBB Version
-        run: pip list | grep openbb
       # This is basically the same as build4mac.sh
       - name: Build Bundle
         run: |
-          conda remove --force sigtool -y
           pyinstaller build/pyinstaller/terminal.spec --clean
           osascript build/pyinstaller/setup_icons.applescript
           mkdir DMG
@@ -102,7 +94,3 @@ jobs:
         run: rm -rf /Users/openbb/Desktop/OpenBB\ Terminal
       - name: Remove OpenBB Exports Folder
         run: rm -rf ~/Desktop/OPENBB-exports
-      - name: Deactivate & Delete Environment
-        run: |
-          conda deactivate
-          conda env remove -n build_env
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -75,6 +75,9 @@ jobs:
       - name: Install Package
         run: poetry install --no-interaction
 
+      - name: Install Optimization Toolkit
+        run: poetry install -E optimization --no-interaction
+
       - name: Run tests
         env:
           MPLBACKEND: Agg
@@ -141,6 +144,9 @@ jobs:
       - name: Install Package
         run: poetry install --no-interaction
 
+      - name: Install Optimization Toolkit
+        run: poetry install -E optimization --no-interaction
+
       - name: List installed packages
         shell: bash -l {0}
         run: |

diff --git a/.github/workflows/windows10_build.yml b/.github/workflows/windows10_build.yml
@@ -32,10 +32,10 @@ jobs:
         run: |
           conda activate obb
           poetry install
-      - name: Poetry Install Predictions
+      - name: Poetry Install Portfolio Optimization and Forecasting Toolkits
         run: |
           conda activate obb
-          poetry install -E prediction
+          poetry install -E all
       - name: Poetry Install Installer
         run: |
           conda activate obb
@@ -49,11 +49,10 @@ jobs:
       - name: Build Terminal.spec file
         run: |
           conda activate obb
-          conda remove --force sigtool -y
           pyinstaller build/pyinstaller/terminal.spec --clean
       - name: Move Files into App Folder
         run: cp -r .\dist\OpenBBTerminal\ .\build\nsis\app\
-      # dont need this because log collection is turned off
+      # don't need this because log collection is turned off
       # - name: Remove Log Data
       #   run: Remove-Item .\build\nsis\app\openbb_terminal\logs\ -Recurse
       - name: Build Exe File

diff --git a/.secrets.baseline b/.secrets.baseline
@@ -1,5 +1,5 @@
 {
-  "version": "1.1.0",
+  "version": "1.4.0",
   "plugins_used": [
     {
       "name": "ArtifactoryDetector"
@@ -20,6 +20,16 @@
     {
       "name": "CloudantDetector"
     },
+    {
+      "name": "DiscordBotTokenDetector"
+    },
+    {
+      "name": "GitHubTokenDetector"
+    },
+    {
+      "name": "HexHighEntropyString",
+      "limit": 3.0
+    },
     {
       "name": "IbmCloudIamDetector"
     },
@@ -31,7 +41,7 @@
     },
     {
       "name": "KeywordDetector",
-      "keyword_exclude": ""
+      "keyword_exclude": "example"
     },
     {
       "name": "MailchimpDetector"
@@ -42,6 +52,9 @@
     {
       "name": "PrivateKeyDetector"
     },
+    {
+      "name": "SendGridDetector"
+    },
     {
       "name": "SlackDetector"
     },
@@ -92,8 +105,26 @@
     },
     {
       "path": "detect_secrets.filters.heuristic.is_templated_secret"
+    },
+    {
+      "path": "detect_secrets.filters.regex.should_exclude_file",
+      "pattern": [
+        "test*.py"
+      ]
+    },
+    {
+      "path": "detect_secrets.filters.regex.should_exclude_line",
+      "pattern": [
+        ".*example.*"
+      ]
+    },
+    {
+      "path": "detect_secrets.filters.regex.should_exclude_secret",
+      "pattern": [
+        "example"
+      ]
     }
   ],
   "results": {},
-  "generated_at": "2021-05-03T16:53:30Z"
+  "generated_at": "2022-11-16T18:07:51Z"
 }
diff --git a/TROUBLESHOOT.md b/TROUBLESHOOT.md
@@ -1,36 +1,31 @@
 # TROUBLESHOOT
 
+<!-- markdownlint-disable MD033 -->
+
 If you are visiting this page it means that you're having issues installing. We deeply apologize for that.
 
-Since the start of the project we've come across different types of issues experienced by the users. This page tries to
-combine issues, and their solutions. This will allow to give the best install experience to everyone regardless of any
-programming skills.
+Since the start of the project we've come across different types of issues experienced by the users. This page tries to combine issues, and their solutions.
+This will allow to give the best install experience to everyone regardless of any programming skills.
 
 <ol>
 <li>
   <a href="#">Standard Install Troubles</a>
   <ul>
     <li><a href="#microsoft-visual-v++">Microsoft Visual C++</a></li>
     <li><a href="#wheel">Wheel</a></li>
-    <li><a href="#cvxpy">Cvxpy</a></li>
-    <li><a href="#numpy">Numpy</a></li>
     <li><a href="#Poetry">Poetry</a></li>
   </ul>
 </li>
 <li>
   <a href="#">ModuleNotFoundError Trouble</a>
   <ul>
     <li><a href="#general">General</a></li>
-    <li><a href="#pypfopt">pypfopt</a></li>
-    <li><a href="#dotenv">dotenv</a></li>
-    <li><a href="#ally">ally</a></li>
   </ul>
 </li>
 <li>
   <a href="#machine-learning-troubles">Machine Learning Troubles</a>
 </li>
 <li>
-  <a href="#other-issues">Other Issues</a>
   <ul>
     <li><a href="#CRLF-versus-LF">CRLF versus LF</a></li>
   </ul>
@@ -41,26 +36,14 @@ programming skills.
 
 ### Microsoft Visual C++
 
-If your terminal has too many red error lines, it is likely that this is the issue.
-Go to this page and install the 2019 Build Tools (not the latest):
+If you are on Windows and your terminal has too many red error lines, it is likely that
+this is the issue. Go to this page and install the 2019 Build Tools (not the latest):
 <https://answers.microsoft.com/en-us/windows/forum/windows_other-windows_install/microsoft-visual-c-140/6f0726e2-6c32-4719-9fe5-aa68b5ad8e6d>
 
 ### Wheel
 
 `conda install -c conda-forge wheel` or `pip install wheel`
 
-### Cvxpy
-
-```bash
-conda install -c conda-forge cvxpy
-```
-
-### Numpy
-
-```bash
-pip install --upgrade numpy==1.20.2
-```
-
 ### Poetry
 
 If you get errors about .whl files not existing (usually on Windows) you have to reinitialize the following folder.
@@ -79,7 +62,7 @@ If you run into trouble with Poetry, and the advice above did not help, your bes
 
 1. `poetry update --lock`
 
-2. `conda deactivate` -> `conda activate gst`, then try again
+2. `conda deactivate` -> `conda activate obb`, then try again
 
 3. Track down the offensive package and purge it from your anaconda `<environment_name>` folder, then try again
    (removing through conda can sometimes leave locks behind)
@@ -112,57 +95,31 @@ as advised [here](https://github.com/python-poetry/poetry/issues/4210) and it sh
 
 ### General
 
-In the case when you run into an error of the form `ModuleNotFoundError: No module named '_______'`. The solution is to
-install the missing package via pip.
-
-If you get the error that `yfinance` is not found, you would run
-
-- `pip install yfinance`
-
-Then please submit an issue so that we can address why that was not imported.
-
-Please note that the package `pmdarima` needs to installed through `pip install` and not through `conda install`.
+In the case when you run into an error of the form `ModuleNotFoundError: No module named '_______'` before you start installing these modules that have not been found please check that you have most followed the recommended installation instructions.
+These errors often can occur when you have not activated the virtual environment where you have installed the terminal, or you have not used the `poetry install` command to install the dependencies.
 
-### pypfopt
-
-```bash
-pip install PyPortfolioOpt
-```
-
-### dotenv
-
-```bash
-pip install python-dotenv
-```
-
-### ally
-
-```bash
-pip install pyally
-```
+In case you wish to proceed with an alternative way to install the terminal feel free to install the missing packages via pip. For example if you get the error that `yfinance` is not found, you would run `pip install yfinance`
 
 ### Machine Learning Troubles
 
 If you run into issues installing or `Cannot convert a symbolic Tensor...` at runtime, try this:
 
 ```bash
-conda install -c conda-forge numpy=1.19.5 hdf5=1.10.5
 poetry install
 poetry install -E prediction
 ```
 
 _Commands that may help you in case of an error:_
 
 - `python -m pip install --upgrade pip`
-- `pip install pystan --upgrade`
 - `poetry update --lock`
-
-## Other Issues
+- `poetry install`
 
 ### CRLF versus LF
 
 When trying to commit code changes, pylint will prevent you from doing so if your line break settings are set to
-CRLF (default for Windows). This is because the entire package uses LF (default for Linux/Mac), and it is therefore
+CRLF (default for Windows).
+This is because the entire package uses LF (default for Linux/Mac), and it is therefore
 important that you change this setting to LF _before_ you make any changes to the code.
 
 It is possible that CRLF automatically turns back on, you can correct this with:
@@ -179,7 +136,7 @@ git rm --cached -r .
 git reset --hard
 ```
 
-### Unable to run gst from VS Code integrated terminal
+### Unable to run openbb from VS Code integrated terminal
 
 Occurs when vscode terminal python version/path is different from the terminal version.