Skip to content

#42: Upgrading Hadoop,Spark,Zookeeper and removing Yarnspawner bug #43

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
GeorgSchulz merged 1 commit into from
Aug 4, 2024
Merged

#42: Upgrading Hadoop,Spark,Zookeeper and removing Yarnspawner bug #43

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

- communitylab#37: Upgrading to Apache Zookeeper 3.8.4
- communitylab#36: Upgrading to Apache Spark 3.5.1
- communitylab#35: Upgrading to Apache Hadoop 3.4.0
- communitylab#26: Error handling in Conda Package Manager

### Added

- communitylab#35: Upgrading to Apache Hadoop 3.4.0
- communitylab#37: Upgrading to Apache Zookeeper 3.9.2
- communitylab#36: Upgrading to Apache Spark 3.5.1
- communitylab#42: Removing bugs in Yarnspawner and upgrading JupyterHub

## [v1.3.0] - 2024-08-02

### Added
Expand Down
8 changes: 4 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,11 @@ Following components are used:

| Component | Version |
|------------------|---------|
| JupyterHub | 4.1.5 |
| JupyterHub | 5.1.0 |
| JupyterLab | 4.2.4 |
| Apache Hadoop | 3.3.4 |
| Apache Spark | 3.1.2 |
| Apache Zookeeper | 3.8.0 |
| Apache Hadoop | 3.4.0 |
| Apache Spark | 3.5.1 |
| Apache Zookeeper | 3.9.2 |
| PostgreSQL | 16 |

The conda environments used for JupyterHub and JupyterLab contain following packages:
Expand Down
4 changes: 2 additions & 2 deletions collections/ansible_collections/bigdata/spark/roles/common/defaults/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
spark_version: "3.1.2"
spark_hadoop_version: "3.2"
spark_version: "3.5.1"
spark_hadoop_version: "3"
hdfs_user: "{{ hdfs_user | default('hdfs') }}"
hdfs_group: "{{ hdfs_group | default('hdfs') }}"
yarn_user: "{{ yarn_user | default('yarn') }}"
Expand Down
2 changes: 1 addition & 1 deletion collections/ansible_collections/bigdata/zookeeper/roles/cluster/defaults/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
zookeeper_version: "3.8.0"
zookeeper_version: "3.9.2"
zookeeper_data_dir: "/var/zookeeper"
zookeeper_log_dir: "/var/log/zookeeper"
10 changes: 5 additions & 5 deletions collections/ansible_collections/jupyter/hub/roles/install/defaults/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
miniforge_base_path: "/opt/miniforge/miniforge"
environment_python_version: "3.9.0"
environment_python_version: "3.10.14"

jupyterhub_conda_packages:
- "jupyterhub=4.1.5"
- "nodejs=17.8.0"
- "configurable-http-proxy=4.5.1"
- "jupyterhub=5.1.0"
- "nodejs=22.5.1"
- "configurable-http-proxy=4.6.2"
- "jupyterhub-yarnspawner=0.4.0"
jupyterhub_pip_packages:
- "git+https://github.com/jupyterhub/yarnspawner.git"
- "git+https://github.com/jupyterhub/ldapauthenticator.git"
- "psycopg2-binary==2.9.9"
90 changes: 54 additions & 36 deletions collections/ansible_collections/jupyter/hub/roles/install/files/jupyterhubenvironment.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,101 +4,119 @@
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
alembic 1.13.2 pyhd8ed1ab_0 conda-forge
async_generator 1.10 py_0 conda-forge
attrs 23.2.0 pyh71513ae_0 conda-forge
annotated-types 0.7.0 pyhd8ed1ab_0 conda-forge
arrow 1.3.0 pyhd8ed1ab_0 conda-forge
async_generator 1.10 pyhd8ed1ab_1 conda-forge
attrs 24.1.0 pyh71513ae_0 conda-forge
blinker 1.8.2 pyhd8ed1ab_0 conda-forge
brotli-python 1.1.0 py39h3d6467e_1 conda-forge
brotli-python 1.1.0 py310hc6cd4ac_1 conda-forge
bzip2 1.0.8 h4bc722e_7 conda-forge
c-ares 1.32.3 h4bc722e_0 conda-forge
ca-certificates 2024.7.4 hbcca054_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
certifi 2024.7.4 pyhd8ed1ab_0 conda-forge
certipy 0.1.3 py_0 conda-forge
cffi 1.16.0 py39h7a31438_0 conda-forge
cffi 1.16.0 py310h2fee648_0 conda-forge
charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge
configurable-http-proxy 4.5.1 node17_h7e777a6_1 conda-forge
cryptography 42.0.8 py39h8169da8_0 conda-forge
greenlet 3.0.3 py39h3d6467e_0 conda-forge
grpcio 1.65.2 pypi_0 pypi
configurable-http-proxy 4.6.2 hbf95b10_1 conda-forge
cryptography 43.0.0 py310h4909e49_0 conda-forge
fqdn 1.5.1 pyhd8ed1ab_0 conda-forge
greenlet 3.0.3 py310hc6cd4ac_0 conda-forge
grpcio 1.62.2 py310h1b8f574_0 conda-forge
h2 4.1.0 pyhd8ed1ab_0 conda-forge
hpack 4.0.0 pyh9f0ad1d_0 conda-forge
hyperframe 6.0.1 pyhd8ed1ab_0 conda-forge
icu 70.1 h27087fc_0 conda-forge
icu 73.2 h59595ed_0 conda-forge
idna 3.7 pyhd8ed1ab_0 conda-forge
importlib-metadata 8.2.0 pyha770c72_0 conda-forge
importlib_resources 6.4.0 pyhd8ed1ab_0 conda-forge
isoduration 20.11.0 pyhd8ed1ab_0 conda-forge
jinja2 3.1.4 pyhd8ed1ab_0 conda-forge
jsonpointer 3.0.0 py310hff52083_0 conda-forge
jsonschema 4.23.0 pyhd8ed1ab_0 conda-forge
jsonschema-specifications 2023.12.1 pyhd8ed1ab_0 conda-forge
jupyter_telemetry 0.1.0 pyhd8ed1ab_1 conda-forge
jupyterhub 4.1.5 pyh31011fe_0 conda-forge
jupyterhub-base 4.1.5 pyh31011fe_0 conda-forge
jsonschema-with-format-nongpl 4.23.0 hd8ed1ab_0 conda-forge
jupyter_events 0.10.0 pyhd8ed1ab_0 conda-forge
jupyterhub 5.1.0 pyh31011fe_0 conda-forge
jupyterhub-base 5.1.0 pyh31011fe_0 conda-forge
jupyterhub-ldapauthenticator 1.3.3.dev0 pypi_0 pypi
jupyterhub-yarnspawner 0.4.0+2.g1c40f52 pypi_0 pypi
jupyterhub-yarnspawner 0.4.0 py310hff52083_2 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
krb5 1.21.3 h659f571_0 conda-forge
ld_impl_linux-64 2.40 hf3520f5_7 conda-forge
ldap3 2.9.1 pypi_0 pypi
libcurl 8.8.0 hca28451_1 conda-forge
libabseil 20240116.2 cxx17_he02047a_1 conda-forge
libcurl 8.9.1 hdb1bdb2_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 hd590300_2 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 14.1.0 h77fa898_0 conda-forge
libgomp 14.1.0 h77fa898_0 conda-forge
libgrpc 1.62.2 h15f2491_0 conda-forge
libnghttp2 1.58.0 h47da74e_1 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libprotobuf 4.25.3 h08a7969_0 conda-forge
libre2-11 2023.09.01 h5a48ba9_2 conda-forge
libsqlite 3.46.0 hde9e2c9_0 conda-forge
libssh2 1.11.0 h0841786_0 conda-forge
libstdcxx-ng 14.1.0 hc0a3c3a_0 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libuv 1.43.0 h7f98852_0 conda-forge
libuv 1.48.0 hd590300_0 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libzlib 1.2.13 h4ab18f5_6 conda-forge
libzlib 1.3.1 h4ab18f5_1 conda-forge
mako 1.3.5 pyhd8ed1ab_0 conda-forge
markupsafe 2.1.5 py39hd1e30aa_0 conda-forge
markupsafe 2.1.5 py310h2372a71_0 conda-forge
ncurses 6.5 h59595ed_0 conda-forge
nodejs 17.8.0 h8839609_1 conda-forge
nodejs 22.5.1 h6d9b948_0 conda-forge
oauthlib 3.2.2 pyhd8ed1ab_0 conda-forge
openssl 3.3.1 h4bc722e_2 conda-forge
packaging 24.1 pyhd8ed1ab_0 conda-forge
pamela 1.1.0 pyh1a96a4e_0 conda-forge
pip 24.2 pyhd8ed1ab_0 conda-forge
pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge
prometheus_client 0.20.0 pyhd8ed1ab_0 conda-forge
protobuf 5.27.3 pypi_0 pypi
psutil 6.0.0 py39hd3abc70_0 conda-forge
protobuf 4.25.3 py310ha8c1f0e_0 conda-forge
psutil 6.0.0 py310hc51659f_0 conda-forge
psycopg2-binary 2.9.9 pypi_0 pypi
pyasn1 0.6.0 pypi_0 pypi
pycparser 2.22 pyhd8ed1ab_0 conda-forge
pycurl 7.45.3 py39had14f81_1 conda-forge
pyjwt 2.8.0 pyhd8ed1ab_1 conda-forge
pyopenssl 24.2.1 pyhd8ed1ab_0 conda-forge
pycurl 7.45.3 py310h80730c7_1 conda-forge
pydantic 2.8.2 pyhd8ed1ab_0 conda-forge
pydantic-core 2.20.1 py310h42e942d_0 conda-forge
pyjwt 2.9.0 pyhd8ed1ab_1 conda-forge
pyopenssl 24.2.1 pyhd8ed1ab_2 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.9.19 h0755675_0_cpython conda-forge
python 3.10.14 hd12c33a_0_cpython conda-forge
python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge
python-json-logger 2.0.7 pyhd8ed1ab_0 conda-forge
python_abi 3.9 4_cp39 conda-forge
pyyaml 6.0.1 pypi_0 pypi
python_abi 3.10 4_cp310 conda-forge
pyyaml 6.0.1 py310h2372a71_1 conda-forge
re2 2023.09.01 h7f4b329_2 conda-forge
readline 8.2 h8228510_1 conda-forge
referencing 0.35.1 pyhd8ed1ab_0 conda-forge
requests 2.32.3 pyhd8ed1ab_0 conda-forge
rpds-py 0.19.1 py39h5cde264_0 conda-forge
ruamel.yaml 0.18.6 py39hd1e30aa_0 conda-forge
ruamel.yaml.clib 0.2.8 py39hd1e30aa_0 conda-forge
setuptools 71.0.4 pyhd8ed1ab_0 conda-forge
rfc3339-validator 0.1.4 pyhd8ed1ab_0 conda-forge
rfc3986-validator 0.1.1 pyh9f0ad1d_0 conda-forge
rpds-py 0.19.1 py310h42e942d_0 conda-forge
setuptools 72.1.0 pyhd8ed1ab_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
skein 0.8.2 pypi_0 pypi
sqlalchemy 2.0.31 py39hd3abc70_0 conda-forge
sqlite 3.46.0 h6d4b2fc_0 conda-forge
skein 0.8.1 py310hff52083_4 conda-forge
sqlalchemy 2.0.31 py310hc51659f_0 conda-forge
tk 8.6.13 noxft_h4845f30_101 conda-forge
tornado 6.4.1 py39hd3abc70_0 conda-forge
tornado 6.4.1 py310hc51659f_0 conda-forge
traitlets 5.14.3 pyhd8ed1ab_0 conda-forge
types-python-dateutil 2.9.0.20240316 pyhd8ed1ab_0 conda-forge
typing-extensions 4.12.2 hd8ed1ab_0 conda-forge
typing_extensions 4.12.2 pyha770c72_0 conda-forge
tzdata 2024a h0c530f3_0 conda-forge
uri-template 1.3.0 pyhd8ed1ab_0 conda-forge
urllib3 2.2.2 pyhd8ed1ab_1 conda-forge
webcolors 24.6.0 pyhd8ed1ab_0 conda-forge
wheel 0.43.0 pyhd8ed1ab_1 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
zipp 3.19.2 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 h4ab18f5_6 conda-forge
zstandard 0.23.0 py39h623c9ba_0 conda-forge
zlib 1.3.1 h4ab18f5_1 conda-forge
zstandard 0.23.0 py310h64cae3c_0 conda-forge
zstd 1.5.6 ha6fb4c9_0 conda-forge
33 changes: 16 additions & 17 deletions collections/ansible_collections/jupyter/lab/roles/setup/defaults/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,34 +1,33 @@
miniforge_base_path: "/opt/miniforge/miniforge"
environment_python_version: "3.9.0"
environment_python_version_short: "3.9"
environment_python_version: "3.10.14"
environment_python_version_short: "3.10"

jupyterlab_conda_packages:
jupyterlab_conda_packages:
# JupyterLab packages
- "conda=23.9.0"
- "conda=24.7.1"
- "nb_conda_kernels=2.5.1"
- "jupyterhub=4.1.5"
- "jupyterhub=5.1.0"
- "jupyterhub-yarnspawner=0.4.0"
- "jupyterlab=4.2.4"
- "jupyterlab-git=0.50.1"
- "nodejs=17.8.0"
- "nodejs=22.5.1"
- "jupyter_conda=5.2.1"
- "bash_kernel=0.9.3"
# Data Science packages - https://builtin.com/data-science/python-libraries-data-science
- "numpy=1.19.5"
- "keras=2.6.0"
- "numpy=1.26.4"
- "keras=3.4.1"
- "pandas=2.2.2"
- "pytorch=1.10.0"
- "scipy=1.13.1"
- "pytorch=2.3.1"
- "scipy=1.14.0"
- "scikit-learn=1.5.1"
- "tensorflow=2.6.2"
- "matplotlib=3.5.3"
- "seaborn=0.12.2"
- "tensorflow=2.16.2"
- "matplotlib=3.9.1"
- "seaborn=0.13.2"
- "theano=1.0.5"
- "pillow=9.4.0"
- "pillow=10.4.0"
- "requests=2.32.3"
- "selenium=4.16.0"
- "selenium=4.23.1"
- "scrapy=2.11.2"
- "pytest=8.3.2"
# Working with Spark
- "pyspark=3.5.1"
jupyterlab_pip_packages:
- "git+https://github.com/jupyterhub/yarnspawner.git"
9 changes: 4 additions & 5 deletions collections/ansible_collections/jupyter/lab/roles/setup/files/jupyter_labhub.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@

# Borrowed and modified from jupyterhub/batchspawner:
# https://github.com/jupyterhub/batchspawner/blob/d1052385f245a3c799c5b81d30c8e67f193963c6/batchspawner/singleuser.py
class YarnSingleUserNotebookApp(SingleUserLabApp):
class YarnSingleUserLabApp(SingleUserLabApp):
@default('port')
def _port(self):
return random_port()
Expand All @@ -40,10 +40,9 @@ def main(argv=None):
if not os.path.exists('.jupyter'):
os.mkdir('.jupyter')
os.environ[var] = './.jupyter'
for var in ['JUPYTERHUB_OAUTH_ACCESS_SCOPES', 'JUPYTERHUB_OAUTH_SCOPES']:
if os.environ.get(var):
os.environ[var] = json.dumps(var)
return YarnSingleUserNotebookApp.launch_instance(argv)
application_user = os.environ["USER"]
os.environ["JUPYTERHUB_OAUTH_ACCESS_SCOPES"] = f'["access:servers!server={application_user}/", "access:servers!user={application_user}"]'
return YarnSingleUserLabApp.launch_instance(argv)


if __name__ == "__main__":
Expand Down
Loading