Add extra dependencies to docker container (#1073)

* Adding optional packages to poetry and docker containers

* Adding optional packages to poetry and docker containers

* Black Version Rev

* PR Comments

* Fixing PR Comments

* Adding NAPALM as optional dependency

docker/Dockerfile

@@ -25,34 +25,36 @@ HEALTHCHECK --interval=5s --timeout=5s --start-period=5s --retries=1 CMD curl --
 FROM base as dev
 
 # Modify the PATH here because otherwise poetry fails 100% of the time. WAT??
-ENV PATH="${PATH}:/root/.poetry/bin"
+ENV PATH="${PATH}:/root/.local/bin"
 
 # Install development OS dependencies
 # hadolint ignore=DL3008
 RUN apt-get update && \
-    apt-get install --no-install-recommends -y openssl build-essential default-libmysqlclient-dev && \
+    apt-get install --no-install-recommends -y openssl build-essential libxmlsec1-dev libxmlsec1-openssl pkg-config default-libmysqlclient-dev libldap-dev libsasl2-dev && \
     apt-get autoremove -y && \
     apt-get clean all && \
     rm -rf /var/lib/apt/lists/*
 
+    # RUN apt-get update && apt-get install --no-install-recommends -y build-essential libssl-dev libxmlsec1-dev libxmlsec1-openssl pkg-config default-libmysqlclient-dev libldap-dev libsasl2-dev
+
 # Install hadolint for linting Dockerfiles
 RUN curl -Lo /usr/bin/hadolint https://github.com/hadolint/hadolint/releases/download/v2.0.0/hadolint-Linux-x86_64 && \
     chmod +x /usr/bin/hadolint
 
 # Install Poetry manually from GitHub because otherwise it installs its own
 # dependencies globally which may conflict with ours.
 # https://python-poetry.org/docs/#osx-linux-bashonwindows-install-instructions
-RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py -o /tmp/get-poetry.py && \
-    python /tmp/get-poetry.py && \
-    rm -f /tmp/get-poetry.py
+RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py -o /tmp/install-poetry.py && \
+    python /tmp/install-poetry.py && \
+    rm -f /tmp/install-poetry.py
 
 # Poetry shouldn't create a venv as we want global install
 # Poetry 1.1.0 added parallel installation as an option;
 # unfortunately it seems to have some issues with installing/updating "requests" and "certifi"
 # while simultaneously atttempting to *use* those packages to install other packages.
 # For now we disable it.
-RUN /root/.poetry/bin/poetry config virtualenvs.create false && \
-    /root/.poetry/bin/poetry config installer.parallel false
+RUN poetry config virtualenvs.create false && \
+    poetry config installer.parallel false
 
 # Keep the project source code from NAUTOBOT_ROOT
 COPY pyproject.toml poetry.lock /source/
@@ -67,7 +69,7 @@ WORKDIR /source
 # Local source code is mounted as volume into container. This approach does not require
 # to rebuild a container when source code change occurs.
 # -------------------------------------------------------------------------------------
-RUN poetry install --no-root --no-ansi --extras mysql
+RUN poetry install --no-root --no-ansi --extras all
 
 # Generate required dirs and self signed ssl certificates
 RUN mkdir /opt/nautobot /prom_cache && openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 -subj \
@@ -90,22 +92,23 @@ COPY nautobot /source/nautobot
 
 # Build the whl for use in the final container and install for the dev container
 RUN poetry build && \
-    poetry install --no-ansi --extras mysql
+    poetry install --no-ansi --extras all
 
 ################################ Stage: cleaninstall
 # Build an image with required dependencies to pull the installation from
 FROM base as cleaninstall
 
 # hadolint ignore=DL3008
-RUN apt-get update && apt-get install --no-install-recommends -y build-essential libssl-dev
+RUN apt-get update && apt-get install --no-install-recommends -y build-essential libssl-dev libxmlsec1-dev libxmlsec1-openssl pkg-config default-libmysqlclient-dev libldap-dev libsasl2-dev
 
 # pyuwsgi doesn't support ssl so we build it from source
 # https://github.com/nautobot/nautobot/issues/193
 RUN pip install --no-cache-dir --no-binary=pyuwsgi pyuwsgi
 
 COPY --from=dev /source/dist/*.whl /tmp
 
-RUN pip install --no-cache-dir /tmp/nautobot*.whl
+# hadolint ignore=DL3013,SC2102
+RUN for whl in /tmp/nautobot*.whl; do pip install --no-cache-dir ${whl}[all]; done
 
 ################################ Stage: final
 # This image will be the production ready image

examples/s3_static_files/README.md

@@ -9,8 +9,8 @@ Nautobot can use [`django-storages`](https://django-storages.readthedocs.io/en/s
 TLDR:
 
 ```shell
-$ echo django-storages >> $NAUTOBOT_ROOT/local_requirements.txt
-$ pip3 install django-storages
+$ echo "nautobot[remote_storage]" >> $NAUTOBOT_ROOT/local_requirements.txt
+$ pip3 install "nautobot[remote_storage]"
 ```
 
 ## Configuration

nautobot/docs/configuration/authentication/ldap.md

@@ -6,13 +6,10 @@ This guide explains how to implement LDAP authentication using an external serve
 
 ### Install System Packages
 
-!!! danger
-    FIXME(jathan): With `wheel` packages asserted, let's make doubly sure these development libraries even need to be installed anymore.
-
 On Ubuntu:
 
 ```no-highlight
-$ sudo apt install -y libldap2-dev libsasl2-dev libssl-dev
+$ sudo apt install -y libldap-dev libsasl2-dev
 ```
 
 On CentOS:
@@ -32,13 +29,13 @@ Activate the Python virtual environment and install the `django-auth-ldap` packa
 
 ```no-highlight
 $ source /opt/nautobot/bin/activate
-(nautobot) $ pip3 install django-auth-ldap
+(nautobot) $ pip3 install "nautobot[ldap]"
 ```
 
 Once installed, add the package to `local_requirements.txt` to ensure it is re-installed during future rebuilds of the virtual environment:
 
 ```no-highlight
-(nautobot) $ echo django-auth-ldap >> /opt/nautobot/local_requirements.txt
+(nautobot) $ echo "nautobot[ldap]" >> /opt/nautobot/local_requirements.txt
 ```
 
 ## Configuration

nautobot/docs/configuration/authentication/sso.md

@@ -23,10 +23,10 @@ If you are using OpenID Connect or SAML you will also need to install the extra
 
 #### OpenID Connect
 
-For OpenID connect, you'll need to install the `openidconnect` Python extra.
+For OpenID connect, you'll need to install the `sso` Python extra.
 
 ```no-highlight
-$ pip3 install "social-auth-core[openidconnect]"
+$ pip3 install "nautobot[sso]"
 ```
 
 #### SAML
@@ -42,10 +42,10 @@ Install the system dependencies as `root`:
 $ sudo apt install -y libxmlsec1-dev libxmlsec1-openssl pkg-config
 ```
 
-Install the `saml` Python extra as the`nautobot` user.
+Install the `sso` Python extra as the`nautobot` user.
 
 ```no-highlight
-$ pip3 install "social-auth-core[saml]"
+$ pip3 install "nautobot[sso]"
 ```
 
 Please see the SAML configuration guide below for an example of how to configure Nautobot to authenticate using SAML with Google as the identity provider.

nautobot/docs/installation/nautobot.md

@@ -137,7 +137,7 @@ $ pip3 install nautobot
 ```
 
 !!! hint 
-    If you are using MySQL as your database backend, use `pip3 install nautobot[mysql]` to install Nautobot and the `mysqlclient` library together!
+    If you are using MySQL as your database backend, use `pip3 install "nautobot[mysql]"` to install Nautobot and the `mysqlclient` library together!
 
 ### Install MySQL client library
 
@@ -147,7 +147,7 @@ If you are using MySQL as your database server you must install the `mysqlclient
     If you're using a MySQL database, Nautobot **will not work** without this client library. You cannot skip this step.
 
 ```no-highlight
-$ pip3 install mysqlclient
+$ pip3 install "nautobot[mysql]"
 ```
 
 Great! We have `NAUTOBOT_ROOT` ready for use by the `nautobot` user, so let's proceed to verifying the installation.
@@ -211,20 +211,20 @@ We will cover two examples of common optional settings below.
 
 Nautobot provides built-in support for the [NAPALM automation](https://github.com/napalm-automation/napalm/) library, which allows Nautobot to fetch live data from devices and return it to a requester via its REST API. The [`NAPALM_USERNAME`](../../configuration/optional-settings#napalm_username) and [`NAPALM_PASSWORD`](../../configuration/optional-settings#napalm_password) configuration parameters define the credentials to be used when connecting to a device.
 
-To use NAPALM, add `napalm` to your `local_requirements.txt` so that it can be installed and kept up to date:
+To use NAPALM, add `nautobot[napalm]` to your `local_requirements.txt` so that it can be installed and kept up to date:
 
 ```no-highlight
-$ echo napalm >> $NAUTOBOT_ROOT/local_requirements.txt
+$ echo "nautobot[napalm]" >> $NAUTOBOT_ROOT/local_requirements.txt
 ```
 
 ### Remote File Storage
 
 By default, Nautobot will use the local filesystem to store uploaded files. To use a remote filesystem, install the [`django-storages`](https://django-storages.readthedocs.io/en/stable/) library and configure your [desired storage backend](../../configuration/optional-settings/#storage_backend) in `nautobot_config.py`.
 
-To use remote file storage, add `django-storages` to your `local_requirements.txt` so that it can be installed and kept up to date:
+To use remote file storage, add `nautobot[remote_storage]` to your `local_requirements.txt` so that it can be installed and kept up to date:
 
 ```no-highlight
-$ echo django-storages >> $NAUTOBOT_ROOT/local_requirements.txt
+$ echo "nautobot[remote_storage]" >> $NAUTOBOT_ROOT/local_requirements.txt
 ```
 
 ## Prepare the Database