Release/3.2.4 (#5696)

* test

* ultima description (#5541)

* Updating showcase page (#5543)

* updating showcase page

* use sdk instead

* add new bot command autopost docs (#5530)

add new bot command autopost docs to docs.openbb.co

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* Hotfix/ultima improvements (#5553)

* tweaks for monetization

* fix linting issue

* fix: Modify 'load' behavior to append data to existing sheet (#5503)

* fix: Modify 'load' behavior to append data to existing sheet

Details:
- Added logic to detect existing sheet and append data if present.
- There is no direct way to append on the sheet, so I am reading the sheet to get the last row number and then writing after that row number.

Resolves: #5472

* Fixed 'worksheet not found' error during export to an existing sheet

* Changes:
* Now if the sheet exists in the xlsx file it will ask the user if he
  wants to replace/append/create new sheet

---------

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* added platform API integration test to Nightly build (#5571)

* Add columns to `stocks/search` (#5565)

* Add columns to `stocks/search`

isin, cusip, figi, composite_figi, shareclass_figi

* Update test_search[False].txt

* Update test_search[True].txt

* Delete tests/openbb_terminal/stocks/cassettes/test_stocks_helper/test_search.yaml

* Create test_search.yaml

---------

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* updating ultima urls (#5587)

* fix: A value is trying to be set on a copy of a slice from a DataFrame #5582 (#5591)

* Update README.md (#5598)

* Hotfix/econometrics export filename fix (#5508)

* fix: Modify 'load' behavior to append data to existing sheet

Details:
- Added logic to detect existing sheet and append data if present.
- There is no direct way to append on the sheet, so I am reading the sheet to get the last row number and then writing after that row number.

Resolves: #5472

* Added --export option for exporting to a desired file in /econometrics menu

* Added a -f/--file option to provide the filename for export.
Made -t/--type and -f/--file mutually exclusive.
If -f is not given, -t is required and the filename will be generated based on the time and module.

* Revert `helper_funcs.py` to original state

* Made some arguments not required.

* Minor changes asked by @deeleeramone

* No sheet name

* black

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>
Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* FIX #5600 Updated the Twitter logo to X logo (#5601)

* Change Ultima paragraph (#5624)

* change ultima paragraph

* remove old paragraph

* Update CONTRIBUTING.md (#5623)

* mobile line-brake fixed (#5604)

In the mobile view (media max 400px ), the text line "Installation >>" appears broken into smaller segments.

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* Docs/terminalpro and improvement (#5622)

* add first iteration of content for Terminal Pro

* Update index.md

* Update dashboards.md

* Update folders.md

* Update grouping.md

* Update report.md

* Update templates.md

* Update quick-start.md

* Update index.md

* Update index.md

* Update home.md

* Update news.md

* Update index.md

* Update index.md

* Update index.md

* Update data-provider.md

* Update chat-with-widget.md

* Update data-manipulation.md

* Update forecasting.md

* fix: images self closing tags

* fix: only pro visitors can see pro

* inter font

* improve SEO content and restructuring

* add new package for toggle on tutorials

* move folder around

* improve main page of each prod

* small updates and improvements

* fix logo above

* small improvement in headtitle for portfolio funcs

* fix typos from OpenAI's GPT-4

* fix _category_ json

* remove generic SEO words

* add script that generates SEO for documentation

* add missing file

* remove sdk warning message

* fix links for andrew

---------

Co-authored-by: jose-donato <zmcdonato@gmail.com>
Co-authored-by: jose-donato <43375532+jose-donato@users.noreply.github.com>
Co-authored-by: andrewkenreich <andrew.kenreich@gmail.com>

* Add light blue banner, blue when clicking sidebar + fix link in docusaurus.config.js (#5627)

* typo

* pro docs tab coloring

* sidebar selection blue

* redirecting missing

* fix docs (#5646)

* Fix the redirect for intro to the correct url for docs (#5658)

* fix the redirect for intro to the correct url for docs

* fix code linting

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* Improve Pro documentation further (#5641)

* small improvement

* Fix TutorialVideo for mobile view

* main page terminal pro docs

* jose whelp

* placeholder index file that needs to be removed

* fix: fixes sidebar, removes unused index.md

* update link to api keys

* update api keys link

* fix hyperlinks in terminal helper

* fix a few more links

* fix some more links

* linter

* black linter

* improve main docs page for each product

* small refactor

* header to take to main page of product

* run OpenAI script to generate SEO for pro content

---------

Co-authored-by: jose-donato <43375532+jose-donato@users.noreply.github.com>

* Update folders.md

* Update integrate-your-own-backend.md

* Update index.md

* Update settings.md (#5665)

* fix front matter (#5667)

* #5633 adding comma every 1000 (#5652)

* #5633 adding comma every 1000

* pylint

---------

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* Hotfix/reports model render report sanitize inputs (#5669)

* Adding more robust special character handling to create_output_path().��Addresses #5299

* Running black linter on code to conform to OBB style guides.

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>
Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* hotfix/fix-gov-histcont: Fix #5650 - /stocks/gov/histcont (#5676)

* fix histcont

* some cleanup

* remove sort line

* 12-31 not 12-12

* columns_keep_types

* remove space in regex (#5675)

* Pin openai (#5685)

* Clarified stocks/ta empty recom object message (#5468)

* Clarified stocks/ta empty recom object message

* black

* lint

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* improved market cap value extraction (#5169)

* improved market cap value extraction

* pylint

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* 3.2.3->3.2.4

* Changelogs

* new changelog

---------

Co-authored-by: Henrique Joaquim <h.joaquim@campus.fct.unl.pt>
Co-authored-by: Andrew <andrew.kenreich@gmail.com>
Co-authored-by: Adi Sai <adithya.sairam1@gmail.com>
Co-authored-by: Sandip Saha <sandipsmoto@gmail.com>
Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>
Co-authored-by: Luqman <luqazino@gmail.com>
Co-authored-by: Adwait <111136306+AdwaitSalankar@users.noreply.github.com>
Co-authored-by: Abhishek Amar <66067391+abhishek-amar@users.noreply.github.com>
Co-authored-by: DidierRLopes <dro.lopes@campus.fct.unl.pt>
Co-authored-by: professssor <62937125+professssor@users.noreply.github.com>
Co-authored-by: Farookh Zaheer Siddiqui <129654632+FarukhS52@users.noreply.github.com>
Co-authored-by: MAX SHESTOV <57421753+maxxsh@users.noreply.github.com>
Co-authored-by: jose-donato <zmcdonato@gmail.com>
Co-authored-by: jose-donato <43375532+jose-donato@users.noreply.github.com>
Co-authored-by: Mohamad Fadil <fadil.parves@gmail.com>
Co-authored-by: Ramazan <guvencramazan@outlook.com>
Co-authored-by: joshuabuildsthings <joshuabuildsthings@users.noreply.github.com>
Co-authored-by: wojda <114031148+polooner@users.noreply.github.com>
Co-authored-by: Bim Phomthong <118195204+bimbolimbo@users.noreply.github.com>

.github/workflows/nightly-build.yml

@@ -53,3 +53,12 @@ jobs:
         with:
           workflow: pypi-nightly.yml
           token: ${{ secrets.GITHUB_TOKEN }}
+
+  trigger-api-integration-test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Trigger Platform API Integration Test
+        uses: aurelien-baudet/workflow-dispatch@v2
+        with:
+          workflow: platform-api-integration-test.yml
+          token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/platform-api-integration-test.yml

@@ -0,0 +1,134 @@
+name: API Integration Tests
+
+on:
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  api-integration-tests:
+    runs-on: ubuntu-latest
+    env:
+      SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
+      OPENBB_API_AUTH: true
+      OPENBB_API_USERNAME: "USER"
+      OPENBB_API_PASSWORD: "PASS"
+
+    steps:
+      - name: Slack Notification - Starting
+        uses: act10ns/slack@v1
+        with:
+          status: starting
+          channel: "#workflows"
+          message: Starting API Integration Test...
+        if: always()
+
+      - name: Checkout code
+        uses: actions/checkout@v3
+
+      - name: Checkout specific ref
+        if: github.event_name == 'pull_request'
+        run: git fetch origin ${{ github.event.pull_request.head.ref }} && git checkout FETCH_HEAD
+
+      - name: Setup Python 3.9
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.9"
+          architecture: x64
+
+      - name: Load cached venv
+        id: cached-python-environment
+        uses: actions/cache@v3
+        with:
+          path: runner_env
+          key: runner_env-${{ runner.os }}-v1-${{ hashFiles('**/poetry.lock', '**/dev_install.py') }}
+
+      - name: Install Dependencies
+        if: steps.cached-python-environment.outputs.cache-hit != 'true'
+        run: |
+          python -m venv runner_env
+          source runner_env/bin/activate
+          pip install pytest poetry\
+            ./openbb_platform \
+            ./openbb_platform/providers/alpha_vantage \
+            ./openbb_platform/providers/cboe \
+            ./openbb_platform/providers/quandl \
+            ./openbb_platform/providers/fmp \
+            ./openbb_platform/providers/yfinance \
+            ./openbb_platform/providers/intrinio \
+            ./openbb_platform/extensions/futures \
+            ./openbb_platform/extensions/qa \
+            ./openbb_platform/extensions/ta \
+            ./openbb_platform/extensions/econometrics
+            pip uninstall -y openbb_core
+            pip uninstall -y openbb_provider
+            cd openbb_platform/platform/core
+            pip install -U .
+            cd ../provider/
+            pip install -U .
+            cd ../../../
+            pip install -U pydantic
+
+      - name: Populate System Setting Files
+        run: |
+          mkdir -p ~/.openbb_platform
+          touch ~/.openbb_platform/system_settings.json
+          echo '{"log_collect":false}' > ~/.openbb_platform/system_settings.json
+          cat ~/.openbb_platform/system_settings.json
+
+      - name: Create and Populate user_settings.json from GitHub Secrets
+        run: |
+          touch ~/.openbb_platform/user_settings.json
+          echo '{
+            "credentials": {
+              "benzinga_api_key": ${{ secrets.BENZINGA_API_KEY }},
+              "fmp_api_key": "${{ secrets.FMP_API_KEY }}",
+              "polygon_api_key": "${{ secrets.POLYGON_API_KEY }}",
+              "alpha_vantage_api_key": "${{ secrets.ALPHA_VANTAGE_API_KEY }}",
+              "fred_api_key": "${{ secrets.FRED_API_KEY }}",
+              "intrinio_api_key": "${{ secrets.INTRINIO_API_KEY }}"
+            }
+          }' > ~/.openbb_platform/user_settings.json
+
+      - name: Launch the Uvicorn Process
+        run: |
+          source runner_env/bin/activate
+          pip list
+          uvicorn openbb_core.api.rest_api:app --host 0.0.0.0 --port 8000 --reload &
+
+      - name: Wait for 10 seconds
+        run: sleep 10
+
+      - name: Run API Integration Tests
+        run: |
+          source runner_env/bin/activate
+          pip list
+          cd openbb_platform
+          pytest -m integration
+
+      - name: Cache pip packages
+        uses: actions/cache@v2
+        with:
+          path: ~/.cache/pip
+          key: runner_env-${{ runner.os }}-v1-${{ hashFiles('**/poetry.lock', '**/dev_install.py') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-
+
+      - name: Upload summary to Slack
+        uses: adrey/slack-file-upload-action@master
+        with:
+          token: ${{ secrets.SLACK_API_TOKEN }}
+          initial_comment: "API Integration test summary"
+          title: "Integration test summary"
+          path: summary.txt
+          channel: ${{ secrets.SLACK_CHANNEL_ID }}
+
+      - name: Slack Notification - Success/Failure
+        uses: act10ns/slack@v1
+        with:
+          status: ${{ job.status }}
+          steps: ${{ toJson(steps) }}
+          channel: "#workflows"
+        if: always()

.github/workflows/platform-integration-test.yml

@@ -0,0 +1,96 @@
+name: Integration Tests
+
+on:
+  push:
+    branches:
+      - release/*
+      - main
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  integration-tests:
+    runs-on: ubuntu-latest
+    env:
+      SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
+    steps:
+    - name: Slack Notification - Starting
+      uses: act10ns/slack@v1
+      with:
+        status: starting
+        channel: '#workflows'
+        message: Starting Integration Test...
+      if: always()
+
+    - name: Checkout code
+      uses: actions/checkout@v3
+
+    - name: Checkout specific ref
+      if: github.event_name == 'pull_request'
+      run: git fetch origin ${{ github.event.pull_request.head.ref }} && git checkout FETCH_HEAD
+
+    - name: Setup Python 3.9
+      uses: actions/setup-python@v4
+      with:
+        python-version: "3.9"
+        architecture: x64
+
+    - name: Install Poetry
+      uses: snok/install-poetry@v1
+      with:
+        version: 1.4.0
+        virtualenvs-create: true
+        virtualenvs-in-project: true
+
+    - name: Setup sudo apt installs for ubuntu-latest
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y \
+            libgtk-3-dev \
+            libwebkit2gtk-4.0-dev
+
+    - name: Load cached venv
+      id: cached-poetry-dependencies
+      uses: actions/cache@v3
+      with:
+        path: .venv
+        key: venv-${{ runner.os }}-v1-${{ hashFiles('**/poetry.lock') }}
+
+    - name: Install dependencies
+      if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
+      run: poetry install --no-interaction -E optimization
+
+    - name: Run integration tests
+      id: integration-tests
+      run: |
+        source $VENV
+        python terminal.py -t | tee result.txt
+        grep "================================ Integration Test Summary ================================" result.txt -A100 | tail --bytes=2000 > summary.txt
+        echo  >> summary.txt
+
+    - name: Run Integration Tests Coverage Report
+      id: integration-tests-coverage
+      run: |
+        source $VENV
+        python terminal.py -t --coverage | tee result.txt
+        sed -n '/Integration Coverage Summary/,$p' result.txt >> summary.txt
+
+    - name: Upload summary to Slack
+      uses: adrey/slack-file-upload-action@master
+      with:
+        token: ${{ secrets.SLACK_API_TOKEN }}
+        initial_comment: "Integration test summary"
+        title: "Integration test summary"
+        path: summary.txt
+        channel: ${{ secrets.SLACK_CHANNEL_ID }}
+
+    - name: Slack Notification - Success/Failure
+      uses: act10ns/slack@v1
+      with:
+        status: ${{ job.status }}
+        steps: ${{ toJson(steps) }}
+        channel: '#workflows'
+      if: always()

CONTRIBUTING.md

@@ -222,7 +222,7 @@ Now that we have added the model function getting, we need to specify that this
 
 If you are adding a new function with a new data source, make a new value in the file.  If the data source requires an
 API key, please refer to the guide below for adding them.  Instructions for obtaining the new api key
-should be included in the file `OpenBBTerminal/website/content/terminal/usage/guides/api-keys.md`.
+should be included in the file `OpenBBTerminal/website/content/terminal/usage/data/api-keys.md`.
 
 ### View
 
@@ -524,7 +524,7 @@ Any new features that do not contain unit tests will not be accepted.
 
 ### Open a Pull Request
 
-For starters, you should ensure that you branch is up to date with the `develop` branch. To do that, one can run the following commands:
+For starters, you should ensure that your branch is up to date with the `develop` branch. To do that, one can run the following commands:
 
 ```bash
 git fetch upstream
@@ -582,7 +582,7 @@ CLI :computer: → `_controller.py` :robot: → `_view.py` :art:  &nbs
 | **Item**     | **Description**                                                                                                                                                                                | **Example**                                      |
 | :----------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------- |
 | **CONTEXT**  | Specific instrument _world_ to analyse.                                                                                                                                                        | `stocks`, `crypto`, `economy`                    |
-| **CATEGORY** | Group of similar COMMANDS to do on the instrument <br /> There are the specialized categories, specific to each CONTEXT and there are common categories which are not specific to one CONTEXT. | `due_diligence`, `technical_analysis`, `insider` |
+| **CATEGORY** | Group of similar COMMANDS to do on the instrument <br /> There are specialized categories, specific to each CONTEXT and there are common categories which are not specific to one CONTEXT. | `due_diligence`, `technical_analysis`, `insider` |
 | **COMMAND**  | Operation on one or no instrument that retrieves data in form of string, table or plot.                                                                                                        | `rating`, `supplier`, `sentiment`                |
 
 The following layout is expected: `/<context>/<category>/<command_files>`
@@ -625,7 +625,7 @@ With:
 
 1. Each function should have default values for non critical kwargs
 
-    - Why? It increases code readability and acts as an input example for the functions arguments. This increases the ease of use of the functions through the SDK, but also just generally.
+    - Why? It increases code readability and acts as an input example for the function's arguments. This increases the ease of use of the functions through the SDK, but also just generally.
 
     > Watch out, add default values whenever possible, but take care for not adding mutable default arguments! [More info](https://docs.python-guide.org/writing/gotchas/#mutable-default-arguments)
 
@@ -701,7 +701,7 @@ With:
 
 3. Each function needs to have a docstring explaining what it does, its parameters and what it returns.
 
-    - Why? You can use the function without reading its source code. This improves the developing experience and SDK usage. The SDK factory also can’t handle functions with out docstrings.
+    - Why? You can use the function without reading its source code. This improves the developing experience and SDK usage. The SDK factory also can’t handle functions without docstrings.
 
     <br>
 
@@ -1399,12 +1399,12 @@ In the `_view.py` files it is common having at the end of each function `export_
 Let's go into each of these arguments:
 
 - `export` corresponds to the type of file we are exporting.
-  - If the user doesn't has anything selected, then this function doesn't do anything.
+  - If the user doesn't have anything selected, then this function doesn't do anything.
   - The user can export multiple files and even name the files.
   - The allowed type of files `json,csv,xlsx` for raw data and `jpg,pdf,png,svg` for figures depends on the `export_allowed` variable defined in `parse_known_args_and_warn`.
 - `os.path.dirname(os.path.abspath(__file__))` corresponds to the directory path
   - This is important when `export folder` selected is the default because the data gets stored based on where it is called.
-  - If this is called from a `common` folder, we can use `os.path.dirname(os.path.abspath(__file__)).replace("common", "stocks")` insteaad
+  - If this is called from a `common` folder, we can use `os.path.dirname(os.path.abspath(__file__)).replace("common", "stocks")` instead
 - `"pt"` corresponds to the name of the exported file (+ unique datetime) if the user doesn't provide one
 - `df_analyst_data` corresponds to the dataframe with data.
 - `sheet_name` corresponds to the name of the sheet in the excel file.
@@ -1523,7 +1523,7 @@ class ForexController(BaseController):
 
 In case the user is interested in a **DYNAMIC** list of options which changes based on user's state, then a class method must be defined.
 
-The example below shows the an excerpt from `update_runtime_choices` method in the [`options` controller](/openbb_terminal/stocks/options/options_controller.py).
+The example below shows an excerpt from `update_runtime_choices` method in the [`options` controller](/openbb_terminal/stocks/options/options_controller.py).
 
 ```python
 def update_runtime_choices(self):
@@ -1554,8 +1554,8 @@ This is storing every logged message inside the following location :
 
 Where $HOME is the user home directory, for instance:
 
-- `C:\Users\foo` if your are in Windows and your name is foo
-- `/home/bar/` if you are is macOS or Linux and your name is bar
+- `C:\Users\foo` if you are in Windows and your name is foo
+- `/home/bar/` if you are in macOS or Linux and your name is bar
 
 The user can override this location using the settings key `OPENBB_USER_DATA_DIRECTORY`.
 
@@ -1610,7 +1610,7 @@ becomes
   stocks/NEWS: mais recentes notícias da empresa
 ```
 
-Note: To speed up translation, the team developed a [script](/i18n/help_translation.ipynb) that uses Google translator API to help translating the entire `en.yml` document to the language of choice. Then the output still needs to be reviewed, but this can be an useful bootstrap.
+Note: To speed up translation, the team developed a [script](/i18n/help_translation.ipynb) that uses Google translator API to help translating the entire `en.yml` document to the language of choice. Then the output still needs to be reviewed, but this can be a useful bootstrap.
 
 This is the convention in use for creating a new key/value pair:
 

README.md

@@ -12,32 +12,35 @@
   <img src="https://github.com/codespaces/badge.svg" height="20" />
 </a>
 
-OpenBB is committed to building the future of investment research by focusing on an open-source infrastructure accessible to everyone, everywhere.
+The first financial terminal that is free and fully open source. With over 600 commands, the terminal has access to equity, options, crypto, forex, macro economy, fixed income, alternative datasets, and more.
 
-<p align="center">
+Sign up to the [OpenBB Hub](https://my.openbb.co/login) to use our Windows or MacOS installers to get started.
+
+<a href="https://www.youtube.com/playlist?list=PLXcRIKcHAUP45_ZZCxxvPyXFKHdq9kM8T">
+  <p align="center">Getting started with the OpenBB Terminal</p>
+</a>
 
+<p align="center">
   <p align="center">
-     <a href="https://www.youtube.com/watch?v=fqGPK8OVHLk" rel="OpenBB Terminal Demo">
+     <a href="https://www.youtube.com/playlist?list=PLXcRIKcHAUP45_ZZCxxvPyXFKHdq9kM8T" rel="OpenBB Terminal Demo">
         <img src="images/openbb_terminal_illustration.gif" alt="OpenBB Terminal Illustration" width="100%"/>
      </a>
-     <p align="center">Click on the GIF above for a DEMO</p>
   </p>
-
   <p align="center">
     <a href="https://docs.openbb.co/terminal/installation">
-      <strong>≪  GETTING STARTED</strong>
+      <strong>≪  INSTALL</strong>
     </a>
     &nbsp · &nbsp
-    <a href="https://github.com/OpenBB-finance/OpenBBTerminal/tree/master/CONTRIBUTING.md">
-      <strong>CONTRIBUTING</strong>
+    <a href="https://my.openbb.co/app/terminal/data">
+      <strong>DATA</strong>
     </a>
     &nbsp · &nbsp
-    <a href="https://openbb.co/docs">
-      <strong>DOCUMENTATION</strong>
+    <a href="https://my.openbb.co/app/terminal/features">
+      <strong>SEE FEATURES</strong>
     </a>
     &nbsp · &nbsp
-    <a href="https://docs.openbb.co/terminal">
-      <strong>SEE FEATURES »</strong>
+    <a href="https://github.com/OpenBB-finance/OpenBBTerminal/tree/master/CONTRIBUTING.md">
+      <strong>CONTRIBUTING »</strong>
     </a>
   </p>
 </p>

build/docker/compose.env

@@ -1,2 +1,2 @@
 OPENBBTERMINAL_DOCKER_REGISTRY="ghcr.io"
-OPENBBTERMINAL_DOCKER_RELEASE_VERSION="3.2.3"
+OPENBBTERMINAL_DOCKER_RELEASE_VERSION="3.2.4"