diff --git a/ir/ir-base/readme.md b/ir/ir-base/readme.md new file mode 100644 index 0000000..7d83fca --- /dev/null +++ b/ir/ir-base/readme.md @@ -0,0 +1,25 @@ +# Quick Reference +- **Maintained by**: [Deepnote](https://deepnote.com/) + +## IR Tags +- [`4.0.3`, `4.2.0`] + +## What is Deepnote IR? +Deepnote IR provides official Docker images that are pre-configured for use on [Deepnote](https://deepnote.com). These images include the R programming language installed in specific versions, along with the [IRkernel](https://github.com/IRkernel/IRkernel), which is an R kernel for Jupyter. + +These images are designed to facilitate seamless integration with Deepnote's platform, allowing users to leverage R for data analysis and visualization within Jupyter notebooks. If you find that the provided images do not meet all your requirements, you are encouraged to extend them by building your own custom images on top of these bases. + +## How to Customize Your Image +To create a custom Docker image based on Deepnote's IR images, you can use the following example Dockerfile: + +```dockerfile +FROM deepnote/ir:4.2.0 + +RUN R -e "install.packages('tidyverse', repos='https://cloud.r-project.org', dependencies=TRUE)" +``` + +This example demonstrates how to build upon the `deepnote/ir:4.2.0` image by installing the tidyverse package, a collection of R packages designed for data science. You can modify this Dockerfile to include other R packages or configurations as needed. + +# Additional Information +These images serve as a robust foundation for R-based data science projects within Deepnote, providing the necessary tools and libraries to get started quickly. For more detailed information about Deepnote and how these images can enhance your data science workflows, please visit Deepnote's website. + diff --git a/python/datascience/Dockerfile.datascience b/python/datascience/Dockerfile.datascience index 4e3b8cf..ac08427 100644 --- a/python/datascience/Dockerfile.datascience +++ b/python/datascience/Dockerfile.datascience @@ -2,31 +2,20 @@ ARG PYTHON_VERSION=3.8 ARG CIRCLE_PULL_REQUEST FROM deepnote/python:${PYTHON_VERSION}${CIRCLE_PULL_REQUEST:+-ra-${CIRCLE_PULL_REQUEST##*/}} -RUN pip install --upgrade pip - - # Add the requirements files ADD requirements-3.11+.txt /requirements-3.11+.txt ADD requirements-below-3.11.txt /requirements-below-3.11.txt -ADD constraints.txt /constraints.txt #Determine the Python version and set the version-specifications file ARG PYTHON_VERSION -RUN python -m pip install --upgrade setuptools pip - RUN if [ "$(printf '%s\n' "$PYTHON_VERSION" "3.11" | sort -V | head -n1)" = "3.11" ]; then \ mv "requirements-3.11+.txt" "requirements.txt" \ ; else \ mv "requirements-below-3.11.txt" "requirements.txt" \ ; fi -# Install dependencies -# Combine all RUN commands into one to minimize the number of layers -RUN apt-get update \ - && apt-get install -y --no-install-recommends gcc \ - && pip install --no-cache-dir -r requirements.txt -c constraints.txt \ - && apt-get purge -y gcc \ - && apt-get autoremove -y \ - && apt-get clean \ - && rm -rf /var/lib/apt/lists/* /root/.cache + +RUN python -m venv --system-site-packages ~/venv +RUN . ~/venv/bin/activate \ + && pip install --no-cache-dir -r requirements.txt -c https://tk.deepnote.com/constraints${PYTHON_VERSION}.txt \ No newline at end of file diff --git a/python/datascience/constraints.txt b/python/datascience/constraints.txt deleted file mode 100644 index 013749a..0000000 --- a/python/datascience/constraints.txt +++ /dev/null @@ -1,147 +0,0 @@ -MarkupSafe>=0.9.2 -MarkupSafe>=2.0 -SQLAlchemy<2.0.0,>=0.9.2 -SQLAlchemy>=1.3.0 -altair>=4.2.2,<=5.5 -asttokens>=2.1.0 -asynch>=0.2.2 -attrs>=19.2.0 -backports.functools-lru-cache>=1.2.1 -bcrypt>=3.2 -botocore<1.36.0,>=1.35.10 -botocore<2.0a.0,>=1.33.2 -cachetools<6.0,>=2.0.0 -certifi>=2017.4.17 -cffi>=1.12 -cffi>=1.4.1 -charset-normalizer<4,>=2 -click-plugins>=1.0 -click>=4.0 -clickhouse-driver>=0.1.2 -clickhouse-sqlalchemy>0,<0.4 -click~=8.0 -cligj>=0.5 -cryptography<44,>=41.0.5 -cryptography>=3.3 -db-dtypes>=1.2.0,<1.4 -dill==0.3.8 -dnspython<3.0.0,>=1.16.0 -dnspython>=2.2.1,<2.7 -duckdb>=0.9.2,<0.10.0 -executing>=1.2.0 -fiona>=1.8 -geopandas>=0.11.1,<1.1 -google-api-core!=2.0.*,!=2.1.*,!=2.10.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,!=2.9.*,<3.0.0dev,>=1.34.0 -google-api-core!=2.0.*,!=2.1.*,!=2.10.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,!=2.9.*,<3.0.0dev,>=1.34.1 -google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0 -google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5 -google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.6 -google-auth<3.0.0dev,>=1.25.0 -google-auth<3.0.0dev,>=2.14.1 -google-auth<3.0.dev0,>=2.14.1 -google-auth<3.0dev,>=1.25.0 -google-cloud-bigquery-storage==2.16.2 -google-cloud-bigquery<4.0.0dev,>=3.3.6 -google-cloud-core<3.0.0dev,>=1.6.0 -google-cloud-core<3.0dev,>=1.4.4 -google-cloud-spanner>=3.12.0 -google-crc32c<2.0dev,>=1.0 -google-resumable-media<3.0dev,>=0.6.0 -googleapis-common-protos<2.0.0dev,>=1.56.0 -googleapis-common-protos<2.0.dev0,>=1.56.2 -googleapis-common-protos>=1.5.5 -greenlet!=0.4.17 -grpc-google-iam-v1<1.0.0dev,>=0.12.4 -grpc-interceptor>=0.15.4 -grpcio<2.0.0dev,>=1.44.0 -grpcio>=1.49.1,<2.0.0 -grpcio>=1.62.3 -idna<4,>=2.5 -importlib-metadata>1.0.0 -ipykernel>6.22.0 -jedi>=0.16 -jinja2>=2.11.3,<3.1 -jmespath<2.0.0,>=0.7.1 -jsonschema>=3.0 -numpy>=1.16.6 -numpy>=1.19,<2 -numpy>=1.20.3 -numpy>=1.21,<2 -numpy>=1.21.0 -numpy>=1.23,<2 -numpy>=1.23.2 -numpy>=1.24,<2 -packaging>=17.0 -packaging>=20.0.0 -pandas>=0.18 -pandas>=0.24.2 -pandas>=1.0.0 -pandas>=1.2.5,<2.2 -paramiko>=2.7.2 -paramiko>=3.0.0,<3.6 -parso<0.8.0,>=0.7.0 -pexpect>4.3 -pg8000==1.31.2 -prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30 -proto-plus<2.0.0dev,>=1.22.0 -proto-plus<2.0.0dev,>=1.22.2 -proto-plus<2.0.0dev,>=1.22.3 -protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 -protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0.dev0,>=3.19.5 -protobuf!=3.20.0,!=3.20.1,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0.dev0,>=3.20.2 -protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0dev,>=3.20.2 -protobuf!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0dev,>=3.20.2 -protobuf<6.0.0dev,>=3.19.0 -protobuf>=4.21.6 -psycopg2-binary>=2.9.4,<2.10 -ptyprocess>=0.5 -pyarrow>13.0.0,<=17.0.0 -pyarrow>=10.0.0,<=17.0.0 -pyarrow>=3.0.0 -pyarrow>=5.0.0 -pyasn1-modules>=0.2.1 -pyasn1<0.7.0,>=0.4.6 -pyasn1>=0.1.3 -pyathena==3.9.0 -pygments>=2.4.0 -pymongo>=4.3.2,<4.10 -pymssql>=2.2.9,<2.4 -pymysql>=1.0.2,<1.2 -pynacl>=1.5 -pyproj>=2.6.1.post1 -python-dateutil<3.0.0,>=2.1 -python-dateutil<3.0dev,>=2.7.2 -python-dateutil>=2.8.1 -pytz>=2020.1 -requests<3.0.0.dev0,>=2.18.0 -requests<3.0.0dev,>=2.21.0 -requests>=2.31.0 -rsa<5,>=3.1.4 -s3transfer<0.11.0,>=0.10.0 -sentry-sdk>=1.24,<2.15 -shapely<2,>=1.7 -six>=1.12.0 -six>=1.5 -snowflake-sqlalchemy>=1,<1.7 -sqlalchemy-bigquery>1.6.1,<1.12 -sqlalchemy-dremio>=3.0.3,<3.1 -sqlalchemy-redshift>=0.8.11,<0.9 -sqlalchemy-spanner>=1.6.2,<2.0.0 -sqlalchemy<1.5,>=1.4.24 -sqlalchemy<3.0.0dev,>=1.4.16 -sqlalchemy>=1.1.13 -sqlalchemy>=1.3.24 -sqlparse>=0.4.4 -sqlparse>=0.4.4,<0.6 -sshtunnel==0.4.0 -traitlets>=5 -trino>=0.327.0,<=0.330.0 -typing-extensions>=3.6.4 -typing-extensions>=4 -urllib3!=2.2.0,<3,>=1.25.4 -urllib3<1.27,>=1.25.4 -urllib3<3,>=1.21.1 -urllib3>=1.25.7 -urllib3>=1.26.11 -urllib3>=1.26.9 -zipp>=0.5 diff --git a/python/python/Dockerfile.python3.10 b/python/python/Dockerfile.python3.10 index 2ef64d3..1531d4c 100644 --- a/python/python/Dockerfile.python3.10 +++ b/python/python/Dockerfile.python3.10 @@ -111,7 +111,3 @@ RUN set -eux; \ ln -svT "$src" "/usr/local/bin/$dst"; \ done - -# We create the virtual environment in the home directory in the Dockerfile -# for performance improvement. -RUN python -m venv --system-site-packages ~/venv diff --git a/python/python/Dockerfile.python3.11 b/python/python/Dockerfile.python3.11 index 50d9a85..a5e3f0d 100644 --- a/python/python/Dockerfile.python3.11 +++ b/python/python/Dockerfile.python3.11 @@ -110,7 +110,3 @@ RUN set -eux; \ ln -svT "$src" "/usr/local/bin/$dst"; \ done - -# We create the virtual environment in the home directory in the Dockerfile -# for performance improvement. -RUN python -m venv --system-site-packages ~/venv diff --git a/python/python/Dockerfile.python3.8 b/python/python/Dockerfile.python3.8 index 5f627e8..316731d 100644 --- a/python/python/Dockerfile.python3.8 +++ b/python/python/Dockerfile.python3.8 @@ -109,9 +109,3 @@ RUN set -eux; \ [ ! -e "/usr/local/bin/$dst" ]; \ ln -svT "$src" "/usr/local/bin/$dst"; \ done - - - -# We create the virtual environment in the home directory in the Dockerfile -# for performance improvement. -RUN python -m venv --system-site-packages ~/venv diff --git a/python/python/Dockerfile.python3.9 b/python/python/Dockerfile.python3.9 index b69c43a..93f3e6f 100644 --- a/python/python/Dockerfile.python3.9 +++ b/python/python/Dockerfile.python3.9 @@ -109,8 +109,3 @@ RUN set -eux; \ [ ! -e "/usr/local/bin/$dst" ]; \ ln -svT "$src" "/usr/local/bin/$dst"; \ done - - -# We create the virtual environment in the home directory in the Dockerfile -# for performance improvement. -RUN python -m venv --system-site-packages ~/venv diff --git a/python/readme.md b/python/readme.md new file mode 100644 index 0000000..1fed85d --- /dev/null +++ b/python/readme.md @@ -0,0 +1,31 @@ +# Quick Reference +- **Maintained by**: [Deepnote](https://deepnote.com/) + +# Supported Tags and Respective Dockerfile Links +- [`3.8`, `3.9`, `3.10`, `3.11`](https://github.com/deepnote/environments/tree/main/python/python) +- [`3.8-datascience`, `3.9-datascience`, `3.10-datascience`, `3.11-datascience`](https://github.com/deepnote/environments/blob/main/python/datascience/Dockerfile.datascience) + +# What is Deepnote Python? +Deepnote Python is a set of Docker images tailored for use in the Deepnote platform. These images are based on the official [Python Docker images](https://hub.docker.com/_/python) and include additional binaries and configurations to enhance the user experience within Deepnote. They are designed to simplify development workflows, especially for data science projects, by providing pre-configured environments that are ready to use. + +# How to Use This Image + +## Create Custom Dockerfile for Deepnote Use +To create a custom Dockerfile using Deepnote's Python image, you can start with the following template: + +```dockerfile +ARG PYTHON_VERSION=3.9 +FROM deepnote/python:$PYTHON_VERSION + +#Determine the Python version and set the version-specifications file +ARG PYTHON_VERSION + +COPY requirements.txt ./ +RUN pip install --no-cache-dir -r requirements.txt -c https://tk.deepnote.com/constraints${PYTHON_VERSION}.txt +``` + +This Dockerfile uses the `deepnote/python:3.9` image as a base and installs Python packages specified in requirements.txt. The use of constraints from `https://tk.deepnote.com/constraints.txt` ensures compatibility and stability of package installations. + +# Additional Information +These images are intended to serve as a starting point for creating reproducible and scalable data science environments in Deepnote. They include various utilities and binaries that streamline workflows, such as Jupyter support, data visualization tools, and more. +For more information about Deepnote and how these images can be used to enhance your data science projects, visit Deepnote's website. diff --git a/readme.md b/readme.md deleted file mode 100644 index 036809e..0000000 --- a/readme.md +++ /dev/null @@ -1,51 +0,0 @@ -# Quick reference - * Maintained by: [Deepnote](https://deepnote.com/) - * Where to get help: [Community](https://community.deepnote.com/join?invitation_token=0ba08a2332e8ec002f56f8c1eefdb5bc49af0bae-ff6f0d9b-5045-4511-b6d3-a4fe2595c951) - -## Python image -* [`3.7`, `3.8`, `3.9`, `3.10`](https://github.com/deepnote/environments/blob/main/python/Dockerfile) - -### How to build -``` -docker build -t deepnote/python: --build-arg FROM_PYTHON_VERSION= ./python -``` - -## Conda images -For every Python version (see above), we also build a `-conda` variant, which includes [Miniconda](https://docs.conda.io/en/latest/miniconda.html) installation. -The Dockerfile for this is at [./python/Dockerfile.conda](./python/Dockerfile.conda). - -### How to build -The Conda image is built on top of the `-bullseye` version of our Python image. We don't currently build it in the CI, so you need to build it locally. -``` -docker build --build-arg FROM_PYTHON_VERSION=${PYTHON_VERSION} --build-arg DEBIAN_VERSION=bullseye -t deepnote/python:${PYTHON_VERSION}-bullseye ./python -``` - -``` -docker build --build-arg FROM_PYTHON_VERSION= -t deepnote/python:-conda ./python --file ./python/Dockerfile.conda -``` - -Before pushing and overwritting the tag currently in use in production, please make sure that the image works as expected with our compute dependencies. We've seen case when the image worked locally (you could run `conda`), but it failed in the Deepnote environment with all compute dependencies installed (the `conda` command failed there). - -You can do this by tagging the image with `-test` suffix and running it in Deepnote as custom environment. - -## R image -* [`3.5.2`, `4.0.4`, `4.2.0`](https://github.com/deepnote/environments/blob/main/ir/Dockerfile) - -We also have a variant with number of preinstalled libraries. We build it for all the above mentioned R versions -and the image is available under `deepnote/ir-with-libs`. See [ir-with-libs](./ir-with-libs/) for more details. - -### How to build -``` -docker build -t deepnote/ir: --build-arg R_BASE_VERSION= ./ir -``` - -## GPU image -The image is based on tensorflow docker image and adds packages that are typical part of our other images (`git` most importantly). - -We follow the tagging of `tensorflow` image. So `tensorflow/tensorflow:2.4.1-gpu` becomes `deepnote/tensorflow:2.4.1-gpu`. - -## Continuous integration -Images are build and pushed to DockerHub using CircleCI pipeline. -Not all images are covered by the continuous integration -- check the [config](./.circleci/config.yml) for more details. - -Pull requests are pushed too, but with `-ra-PR_NUMBER` suffix in the tag. \ No newline at end of file