Switch from Gym to Gymnasium (#30)

Close #26 

* Switch from gymnasium to gym
* Make breaking changes as described here: https://gymnasium.farama.org/content/migration-guide/
* Update Jupyter notebooks

---

Additional changes unrelated to gymnasium:

* Update dependencies and remove unnecessary version pins
* Change to style format black and add pre-commit hooks
* Add some basic test for each registered env

.github/workflows/python-package.yml

@@ -7,7 +7,7 @@ on:
     branches:
       - main
   pull_request:
-    branches: 
+    branches:
       - main
   workflow_dispatch:
 

.gitignore

@@ -1,4 +1,5 @@
 .idea
+.vscode
 results*
 
 # Byte-compiled / optimized / DLL files

.pre-commit-config.yaml

@@ -0,0 +1,19 @@
+repos:
+-   repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v2.3.0
+    hooks:
+    -   id: check-yaml
+    -   id: end-of-file-fixer
+    -   id: trailing-whitespace
+-   repo: https://github.com/psf/black
+    rev: 22.10.0
+    hooks:
+    -   id: black
+-   repo: https://github.com/pycqa/flake8
+    rev: 6.0.0
+    hooks:
+    -   id: flake8
+-   repo: https://github.com/pycqa/isort
+    rev: 5.12.0
+    hooks:
+    -   id: isort

README.md

@@ -1,22 +1,23 @@
 [![CI](https://github.com/stefanbschneider/mobile-env/actions/workflows/python-package.yml/badge.svg)](https://github.com/stefanbschneider/mobile-env/actions/workflows/python-package.yml)
 [![PyPI](https://github.com/stefanbschneider/mobile-env/actions/workflows/python-publish.yml/badge.svg)](https://github.com/stefanbschneider/mobile-env/actions/workflows/python-publish.yml)
 [![Documentation](https://readthedocs.org/projects/mobile-env/badge/?version=latest)](https://mobile-env.readthedocs.io/en/latest/?badge=latest)
+[![Code Style: Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/stefanbschneider/mobile-env/blob/master/examples/demo.ipynb)
 
 
 # mobile-env: An Open Environment for Autonomous Coordination in Mobile Networks
 
-mobile-env is an open, minimalist OpenAI Gym environment for training and evaluating coordination algorithms in wireless mobile networks. 
+mobile-env is an open, minimalist OpenAI Gym environment for training and evaluating coordination algorithms in wireless mobile networks.
 The environment allows modeling users moving around an area and can connect to one or multiple base stations.
 Using the Gym interface, the environment can be used with any reinforcement learning framework (e.g., stable-baselines or Ray RLlib) or any custom (even non-RL) coordination approach.
 The environment is highly configurable and can be easily extended (e.g., regarding users, movement patterns, channel models, etc.).
 
 mobile-env supports multi-agent and centralized reinforcement learning policies. It provides various choices for rewards and observations. mobile-env is also easily extendable, so that anyone may add another channel models (e.g. path loss), movement patterns, utility functions, etc.
 
 As an example, mobile-env can be used to study multi-cell selection in coordinated multipoint.
-Here, it must be decided what connections should be established among user equipments (UEs) and base stations (BSs) in order to maximize Quality of Experience (QoE) globally. 
-To maximize the QoE of single UEs, the UE intends to connect to as many BSs as possible, which yields higher (macro) data rates. 
-However, BSs multiplex resources among connected UEs (e.g. schedule physical resource blocks) and, therefore, UEs compete for limited resources (conflicting goals). 
+Here, it must be decided what connections should be established among user equipments (UEs) and base stations (BSs) in order to maximize Quality of Experience (QoE) globally.
+To maximize the QoE of single UEs, the UE intends to connect to as many BSs as possible, which yields higher (macro) data rates.
+However, BSs multiplex resources among connected UEs (e.g. schedule physical resource blocks) and, therefore, UEs compete for limited resources (conflicting goals).
 To maximize QoE globally, the policy must recognize that (1) the data rate of any connection is governed by the channel (e.g. SNR) between UE and BS and (2) QoE of single UEs not necessarily grows linearly with increasing data rate.
 
 <p align="center">
@@ -77,26 +78,27 @@ For dependencies for building docs, install the requirements in `docs`.
 ## Example Usage
 
 ```python
-import gym
+import gymnasium as gym
 import mobile_env
 
 env = gym.make("mobile-medium-central-v0")
-obs = env.reset()
+obs, info = env.reset()
 done = False
 
 while not done:
     action = ... # Your agent code here
-    obs, reward, done, info = env.step(action)
+    obs, reward, terminated, truncated, info = env.step(action)
+    done = terminated or truncated
     env.render()
 ```
 
 ## Customization
 
-mobile-env supports custom channel models, movement patterns, arrival & departure models, resource multiplexing schemes and utility functions. 
+mobile-env supports custom channel models, movement patterns, arrival & departure models, resource multiplexing schemes and utility functions.
 For example, replacing the default [Okumura–Hata](https://en.wikipedia.org/wiki/Hata_model) channel model by a (simplified) path loss model can be as easy as this:
 
 ```python
-import gym
+import gymnasium as gym
 import numpy as np
 from mobile_env.core.base import MComCore
 from mobile_env.core.channel import Channel
@@ -115,7 +117,7 @@ class PathLoss(Channel):
         return loss
 
 
-# replace default channel model in configuration 
+# replace default channel model in configuration
 config = MComCore.default_config()
 config['channel'] = PathLoss
 
@@ -143,5 +145,3 @@ Development: [@stefanbschneider](https://github.com/stefanbschneider) and [@stwe
 We happy if you find `mobile-env` useful. If you have feedback or want to report bugs, feel free to [open an issue](https://github.com/stefanbschneider/mobile-env/issues/new). Also, we are happy to link to your projects if you use `mobile-env`.
 
 We also welcome contributions: Whether you implement a new channel model, fix a bug, or just make a minor addition elsewhere, feel free to open a pull request!
-
-

docs/components.md

@@ -14,7 +14,7 @@ The [base environment](https://mobile-env.readthedocs.io/en/latest/source/mobile
 
 1. [Channel](https://mobile-env.readthedocs.io/en/latest/source/mobile_env.core.html#mobile_env.core.channels.Channel): computes the data rate of connections among UEs and BSs.
 2. [Arrival](https://mobile-env.readthedocs.io/en/latest/source/mobile_env.core.html#mobile_env.core.arrival.Arrival): specifies the arrival and departure times of UEs, i.e., defines when and how long UEs request service.
-3. [Movement](https://mobile-env.readthedocs.io/en/latest/source/mobile_env.core.html#mobile_env.core.movement.Movement): model to define the movement patterns of UEs. 
+3. [Movement](https://mobile-env.readthedocs.io/en/latest/source/mobile_env.core.html#mobile_env.core.movement.Movement): model to define the movement patterns of UEs.
 4. [Schedule](https://mobile-env.readthedocs.io/en/latest/source/mobile_env.core.html#mobile_env.core.schedules.Scheduler): defines how BSs multiplex resources among all connected UEs.
 5. [Utility Function](https://mobile-env.readthedocs.io/en/latest/source/mobile_env.core.html#mobile_env.core.utilities.Utility): defines a function that quantifies the quality of experience (QoE) of UEs dependend on their (macro) data rate.
 
@@ -27,20 +27,20 @@ class LazyUEMovement(RandomWaypointMovement):
         super().__init__(**kwargs)
         # this UE doen't move!
         self.lazy_ue = lazy_ue
-        
+
     def move(self, ue):
         """Overrides default movement pattern."""
         # do not move lazy UE!
         if ue.ue_id == self.lazy_ue:
             return ue.x, ue.y
-        
+
         # default movement otherwise
         return super().move(ue)
 
 
 from mobile_env.core.base import MComCore
 
-# replace default movement in configuration 
+# replace default movement in configuration
 config = MComCore.default_config()
 config['movement'] = LazyUEMovement
 # pass init parameters to custom movement class!
@@ -71,7 +71,7 @@ class PathLoss(Channel):
         return loss
 
 
-# replace default channel model in configuration 
+# replace default channel model in configuration
 config = MComCore.default_config()
 config['channel'] = PathLoss
 
@@ -118,12 +118,12 @@ So far, [mobile-env](https://mobile-env.readthedocs.io/en/latest/index.html) imp
 ## Scenarios
 So far, mobile-env provides three simulation environments from small to large! They mostly differ in terms of the number of basestations and user equipments.
 
-Small Size             |  Medium Size             |  Large Size             
+Small Size             |  Medium Size             |  Large Size
 :-------------------------:|:-------------------------:|:-------------------------:
 ![](./images/small.png)  |  ![](./images/medium.png)  |  ![](./images/large.png)
 
 ## Wrappers
-Wrapper classes are used to make interfaces compatible with other libraries if they aren't already. For example, the multi-agent reinforcement learning libraries [RLlib](https://docs.ray.io/en/stable/rllib.html) and [PettingZoo](https://www.pettingzoo.ml/) expect different interfaces. So far, we only provide a wrapper class for RLlib's MultiAgentEnv, so that RLlib's multi-agent algorithms can be used to train on mobile-env!. 
+Wrapper classes are used to make interfaces compatible with other libraries if they aren't already. For example, the multi-agent reinforcement learning libraries [RLlib](https://docs.ray.io/en/stable/rllib.html) and [PettingZoo](https://www.pettingzoo.ml/) expect different interfaces. So far, we only provide a wrapper class for RLlib's MultiAgentEnv, so that RLlib's multi-agent algorithms can be used to train on mobile-env!.
 
 Example usage of our **RLlibMAWrapper** class:
 ```python
@@ -134,4 +134,4 @@ env = gym.make('mobile-small-ma-v0')
 env = RLlibMAWrapper(env)
 # use RLlib to train on the environment ...
 ...
-```
+```

docs/conf.py

@@ -12,43 +12,44 @@
 #
 import os
 import sys
-sys.path.insert(0, os.path.abspath('../'))
+
+sys.path.insert(0, os.path.abspath("../"))
 
 
 # -- Project information -----------------------------------------------------
 
-project = 'mobile-env'
-copyright = '2021, Stefan Schneider, Stefan Werner'
-author = 'Stefan Schneider, Stefan Werner'
+project = "mobile-env"
+copyright = "2021, Stefan Schneider, Stefan Werner"
+author = "Stefan Schneider, Stefan Werner"
 
 # The full version, including alpha/beta/rc tags
-release = '1.0.0'
+release = "1.0.0"
 
 
 # -- General configuration ---------------------------------------------------
 
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ['myst_parser', 'sphinx.ext.autodoc', 'sphinx.ext.napoleon']
+extensions = ["myst_parser", "sphinx.ext.autodoc", "sphinx.ext.napoleon"]
 
 # Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
+templates_path = ["_templates"]
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This pattern also affects html_static_path and html_extra_path.
-exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
 
 
 # -- Options for HTML output -------------------------------------------------
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = 'sphinx_rtd_theme'
+html_theme = "sphinx_rtd_theme"
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = []
+html_static_path = []

docs/examples.md

@@ -12,25 +12,26 @@ We provide an in-depth example of mobile-env's usage on Google Colab! The notebo
 `mobile-env` follows the OpenAI Gym interface.
 Here is an example of how mobile-env's environments can be created:
 ```python
-import gym
+import gymnasium as gym
 import mobile_env
 
-# small environment; centralized control 
+# small environment; centralized control
 env = gym.make('mobile-small-central-v0')
 
-# large environment; centralized control 
+# large environment; centralized control
 env = gym.make('mobile-large-central-v0')
 
-# small environment; multi-agent control 
+# small environment; multi-agent control
 env = gym.make('mobile-large-ma-v0')
 ...
 
 # then run the environment
-obs = env.reset()
+obs, info = env.reset()
 done = False
 
 while not done:
     action = ... # Your agent code here
-    obs, reward, done, info = env.step(action)
+    obs, reward, terminated, truncated, info = env.step(action)
+    done = terminated or truncated
     env.render()
 ```

docs/requirements.txt

@@ -2,4 +2,4 @@ sphinx==3.5.4
 sphinx-rtd-theme==0.4.3
 myst-parser==0.15.2
 # Ref: https://github.com/readthedocs/readthedocs.org/issues/9038
-jinja2<3.1
+jinja2<3.1

docs/scripts/utility.py

@@ -1,26 +1,32 @@
 import matplotlib.pyplot as plt
 import numpy as np
 
-
 MIN_UTILITY = -20
 MAX_UTILITY = 20
 
 
 def log_utility(curr_dr):
-    """
-    More data rate increases the utility following a log function: High initial increase, then flattens.
+    """More data rate increases the utility following a log function.
+
+     High initial increase, then flattens.
     :param curr_dr: Current data rate
     :param factor: Factor to multiply the log function with
     :param add: Add to current data rate before passing to log function
     :return: Utility
     """
-    # 4*log(0.1+x) looks good: around -10 for no dr; 0 for 0.9 dr; slightly positive for more
-    # 10*log10(0.1+x) is even better because it's steeper, is exactly -10 for dr=0, and flatter for larger dr
-    # with many UEs where each UE only gets around 0.1 data rate, 100*log(0.9+x) looks good (eg, 50 UEs on medium env)
+    # 4*log(0.1+x) looks good: around -10 for no dr; 0 for 0.9 dr;
+    # slightly positive for more
+    # 10*log10(0.1+x) is even better because it's steeper,
+    # is exactly -10 for dr=0, and flatter for larger dr
+    # with many UEs where each UE only gets around 0.1 data rate,
+    # 100*log(0.9+x) looks good (eg, 50 UEs on medium env)
 
-    # better: 10*log10(x) --> clip to [-20, 20]; -20 for <= 0.01 dr; +20 for >= 100 dr
+    # better: 10*log10(x) --> clip to [-20, 20];
+    # -20 for <= 0.01 dr; +20 for >= 100 dr
     # ensure min/max utility are set correctly for this utility function
-    assert MIN_UTILITY == -20 and MAX_UTILITY == 20, "The chosen log utility requires min/max utility to be -20/+20"
+    assert (
+        MIN_UTILITY == -20 and MAX_UTILITY == 20
+    ), "The chosen log utility requires min/max utility to be -20/+20"
     if curr_dr == 0:
         return MIN_UTILITY
     return np.clip(10 * np.log10(curr_dr), MIN_UTILITY, MAX_UTILITY)