feat: persistence support

Guibod · Apr 1, 2023 · 24db834 · 24db834
1 parent 06bb7ab
commit 24db834
Show file tree

Hide file tree

Showing 100 changed files with 83,101 additions and 3,468 deletions.
diff --git a/README.md b/README.md
@@ -6,15 +6,24 @@
 [![security: bandit](https://img.shields.io/badge/security-bandit-yellow.svg)](https://github.com/PyCQA/bandit)
 [![OpenSSF Best Practices](https://bestpractices.coreinfrastructure.org/projects/7037/badge)](https://bestpractices.coreinfrastructure.org/projects/7037)
 
-A library manage all things Magic The Gathering related in python
+A library manage all things Magic The Gathering related in python.
+
+* Mightstone use `Pydantic`, `Beanie` and `Asyncio` as core feature.
+* Integrated persistence support through `Beanie` of many data classes. Download once, and use data offline.
+* HTTP cache integration
+* Supported services:
+  * [Scryfall](https://scryfall.com)
+  * [EDHREC](https://edhrec.com/)
+  * [MTGJSON](https://mtgjson.com/)
+  * [CardConjurer](https://cardconjurer.com/)
+  * [Magic The Gathering](https://magic.wizards.com/en/rules>) (rules)
 
 
 ## Developing
 
 Run `make` for help
 
     make install             # Run `poetry install`
-    make showdeps            # run poetry to show deps
     make lint                # Runs bandit and black in check mode
     make format              # Formats you code with Black
     make test                # run pytest with coverage

diff --git a/docs/source/_autosummary/mightstone.services.mtgjson.models.rst b/docs/source/_autosummary/mightstone.services.mtgjson.models.rst
diff --git a/docs/source/_autosummary/mightstone.services.mtgjson.rst b/docs/source/_autosummary/mightstone.services.mtgjson.rst
diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -12,19 +12,30 @@ You can install mightstone from PyPI using pip:
 
    pip install mightstone
 
+or through Poetry:
+
+.. code::
+
+  poetry add mightstone
+
 
 Features
 ========
 
- * Mightstone use ``Pydantic`` and ``Asyncio`` as core feature.
- * Supported services:
-    * Scryfall API
-    * EDHREC (most features, the datamodel needs to be enhanced)
-    * MTGJSON
- * Magic The Gathering
-    * A ColorPie generator
-    * A robust Color identity Map
-    * A comprehensive rules API
+* Mightstone use ``Pydantic``, ``Beanie`` and ``Asyncio`` as core feature.
+* Integrated persistence support through ``Beanie`` of many data classes. Download once, and use data offline.
+* HTTP cache integration
+* Supported services:
+
+  * `Scryfall <https://scryfall.com>`_
+  * `EDHREC <https://edhrec.com/>`_
+  * `MTGJSON <https://mtgjson.com/>`_
+  * `CardConjurer <https://cardconjurer.com/>`_
+  * `Magic The Gathering <https://magic.wizards.com/en/rules>`_ (rules)
+
+* A ColorPie generator
+* A robust Color identity Map
+* A comprehensive rules API
 
 Plans
 =====
@@ -33,15 +44,10 @@ Plans
     * magicthegathering.io
     * 17lands
     * draftsim
-    * Card Conjurer (https://cardconjurer.com/home)
     * MTGGoldfish
     * MtgTo8
 
  * An API to handle core concepts such as Abilities, ManaCost, Colors and Color Identity.
- * This API should be able to parse and understand w
- * a persistence tool (most probably using Beanie and Mongodb) to allow an offline usage of scryfall or mtgio datas
- * a better documentation
- * a finalized asyncio interface (I need to pick either aiostream or asyncstdlib)
  * I also hope to provide tools for pandas integration
 
 .. toctree::
@@ -50,6 +56,8 @@ Plans
 
    users/installation
    users/cli
+   users/configuration
+   users/persistence
    users/cardconjurer
    users/asyncio
 

diff --git a/docs/source/reference/api.rst b/docs/source/reference/api.rst
@@ -97,7 +97,6 @@ Models
    :toctree: _autosummary
 
    Card
-   CardAtomicGroup
    Set
    SetList
    Meta

diff --git a/docs/source/users/configuration.rst b/docs/source/users/configuration.rst
@@ -0,0 +1,73 @@
+Configuration
+*************
+
+Mightstone behavior is defined by configuration. You can configure your mightstone by three means:
+
+* Programmatically with a ``MightstoneSettings`` instance passed to your ``Mightstone`` application instance
+* Through environment variables, for instance ``MIGHTSTONE_APPNAME`` to alter ``appname`` key
+* Through a configuration file, with 3 means of defining
+
+  * In CLI by passing a ``--config`` parameter to the command line client
+  * Via ``MIGHTSTONE_CONFIG_FILE`` environment variable
+  * Passively, by storing your configuration file in an standard location.
+
+Configuration
+=============
+
+=========================== ======================================  ====== ======================== ===========
+Path                        Environment Variable                    Type   default                  description
+=========================== ======================================  ====== ======================== ===========
+``appname``                 ``MIGHTSTONE_APPNAME``                  string ``mightstone``           Your application name, it will be used to build your storage and cache paths on the filesystem.
+=========================== ======================================  ====== ======================== ===========
+
+
+
+Storage
+-------
+
+=========================== ======================================  ====== ======================== ===========
+Path                        Environment Variable                    Type   default                  description
+=========================== ======================================  ====== ======================== ===========
+``storage.implementation``  ``MIGHTSTONE_STORAGE_IMPLEMENTATION``   string ``beanita``              Either ``beanita`` for a local file base database, ``motor`` for a remote mongodb connection.
+``storage.uri``             ``MIGHTSTONE_STORAGE_URI``              string                          Mongodb dsn (``mongodb://login:password@host/db``...), only used if ``motor`` implementation
+``storage.directory``       ``MIGHTSTONE_STORAGE_DIRECTORY``        string ``<user_data>/mongita``  Data storage directory, only used if ``beanita`` implementation
+``storage.database``        ``MIGHTSTONE_STORAGE_DATABASE``         string ``mightstone``           The mongo (or beanita) database
+=========================== ======================================  ====== ======================== ===========
+
+Http
+----
+
+=========================== ======================================  ============= ========================= ===========
+Path                        Environment Variable                    Type          default                   description
+=========================== ======================================  ============= ========================= ===========
+``http.cache.persist``      ``MIGHTSTONE_CACHE_PERSIST``            bool          ``True``                  Use memory (transient) or filesystem (persistent) cache
+``http.cache.directory``    ``MIGHTSTONE_CACHE_DIRECTORY``          string        ``<user_cache>/http``     The directory to store HTTP cache, only used of ``persist`` is ``true``
+``http.cache.methods``      ``MIGHTSTONE_CACHE_METHODS``            list[string]  ``[GET]``                 HTTP verbs to cache
+``http.cache.status``       ``MIGHTSTONE_CACHE_STATUS``             list[int]     ``[200,203,300,301,308]`` HTTP status code to cache
+=========================== ======================================  ============= ========================= ===========
+
+
+Where to store your configuration ?
+===================================
+
+Mightstone will search configuration file in this order:
+
+* ``<current_directory>/mightstone.yaml``
+* ``<current_directory>/mightstone.yml``
+* ``<current_directory>/mightstone.json``
+* ``<current_directory>/mightstone.toml``
+* ``<user_config_directory>/mightstone.yaml``
+* ``<user_config_directory>/mightstone.yml``
+* ``<user_config_directory>/mightstone.json``
+* ``<user_config_directory>/mightstone.toml``
+
+Mighstone relies on `AppDirs <https://github.com/ActiveState/appdirs>`_ package to resolve user config directory:
+
+::
+
+    Mac OS X:               ~/Library/Application Support/Mightstone
+    Unix:                   ~/.config/Mightstone     # or in $XDG_CONFIG_HOME, if defined
+    Win XP (not roaming):   C:\Documents and Settings\<username>\Application Data\Mightstone\Mightstone
+    Win XP (roaming):       C:\Documents and Settings\<username>\Local Settings\Application Data\Mightstone\Mightstone
+    Win 7  (not roaming):   C:\Users\<username>\AppData\Local\Mightstone\Mightstone
+    Win 7  (roaming):       C:\Users\<username>\AppData\Roaming\Mightstone\Mightstone
diff --git a/docs/source/users/persistence.rst b/docs/source/users/persistence.rst
@@ -0,0 +1,25 @@
+Persistence
+***********
+
+Mightstone uses `Beanie <https://beanie-odm.dev/>`_ as storage backend. This implies that you need to setup beanie backend before any instance of ``MightstoneDocument`` is created.
+
+``beanie.exceptions.CollectionWasNotInitialized`` exception are raised as soon as a ``MightstoneDocument`` class is instanced before ``init_beanie`` is called. Please check `Beanie Documentation <https://beanie-odm.dev/tutorial/initialization/>`_ to learn more.
+
+Automatic setup (recommended)
+=============================
+
+If you access to Mightstone content through the a ``Mightstone()`` this is done automatically.
+
+.. literalinclude:: ../../../examples/persistence.py
+   :language: python
+
+This approach is recommended since our beanie implementation may evolved drastically.
+
+Manual setup
+============
+
+If you wish to use base object, you’ll need to initialize beanie.
+
+.. literalinclude:: ../../../examples/persistence_minimal.py
+   :language: python
+
diff --git a/examples/card_conjurer.py b/examples/card_conjurer.py
@@ -1,16 +1,18 @@
 import asyncio
 
-from mightstone.services.cardconjurer import CardConjurer
+from mightstone.app import Mightstone
 
-cc = CardConjurer()
 
-
-async def run():
-    card = await cc.card("my_card.json")
+async def run(mightstone: Mightstone):
+    cc = mightstone.card_conjurer
+    card = await cc.card_async(
+        "../tests/services/cardconjurer/samples/Dimirova Smiley.json"
+    )
     # You might need to define the remote server if the card was
     # remotely built and no asset are present locally
     card.asset_root_url = "https://card-conjurer-assets.s3.us-east-1.amazonaws.com"
-    await cc.render(card, "my_card.png")
+    await cc.render_async(card, "my_card.png")
 
 
-asyncio.run(run())
+mightstone = Mightstone()
+asyncio.run(run(mightstone))
diff --git a/examples/comprehensive_rules.py b/examples/comprehensive_rules.py
@@ -4,16 +4,16 @@
 
 from pydantic.json import pydantic_encoder
 
-from mightstone.services.wotc import RuleExplorer
+from mightstone.app import Mightstone
 from mightstone.services.wotc.models import RuleRef, RuleText
 
-explorer = RuleExplorer()
-before_errata = explorer.open(
+mightstone = Mightstone()
+before_errata = mightstone.rule_explorer.open(
     "https://media.wizards.com/2020/downloads/MagicCompRules%2020200417.txt"
 )
 
 errata_companion = asyncio.run(
-    explorer.open_async(
+    mightstone.rule_explorer.open_async(
         "https://media.wizards.com/2020/downloads/MagicCompRules%2020200601.txt"
     )
 )
@@ -28,7 +28,11 @@
 
 # Brute force find all rules between January 1st, 2022 and  February 15, 2023
 # using a maximum of 10 requests at a time
-print(explorer.explore(date(2020, 1, 1), date(2023, 2, 15), concurrency=10))
+print(
+    mightstone.rule_explorer.explore(
+        date(2020, 1, 1), date(2023, 2, 15), concurrency=10
+    )
+)
 
 # 'https://media.wizards.com/2020/downloads/MagicCompRules%2020200122.txt',
 # 'https://media.wizards.com/2020/downloads/MagicCompRules%2020200417.txt',
@@ -52,20 +56,20 @@
 # 'https://media.wizards.com/2023/downloads/MagicComp%20Rules%2020230203.txt'
 
 # Read the latest comprehensive rules
-latest_url = explorer.latest()
+latest_url = mightstone.rule_explorer.latest()
 print(latest_url)
 
-cr = explorer.open(latest_url)
+cr = mightstone.rule_explorer.open(latest_url)
 
 # Or simply
-# cr = explorer.open()
+# cr = mightstone.rule_explorer.open()
 
 # Or from an URL
-# cr = explorer.open(
+# cr = mightstone.rule_explorer.open(
 #    "https://media.wizards.com/2022/downloads/MagicCompRules%2020221118.txt")
 #
 # Or from a local file
-# cr = explorer.open("/path/to/comprehensive-rules.txt")
+# cr = mightstone.rule_explorer.open("/path/to/comprehensive-rules.txt")
 
 # Access effectiveness date (cr.effective is an Effectiveness object)
 print(cr.effective.date)  # 2023-02-03

diff --git a/examples/edhrec.py b/examples/edhrec.py
@@ -1,13 +1,13 @@
 import asyncio
 
-from mightstone.services.edhrec import EdhRecStatic
+from mightstone.app import Mightstone
 
-edhrec = EdhRecStatic()
+mightstone = Mightstone()
 
 
 async def async_sample():
     items = []
-    async for companion in edhrec.companions_async():
+    async for companion in mightstone.edhrec_static.companions_async():
         items.append(companion)
     return items
 

diff --git a/examples/mightstone.yml b/examples/mightstone.yml
@@ -0,0 +1,13 @@
+appname: my-stone
+
+storage:
+  implementation: beanita
+  directory: /tmp/my-stone/data
+  database: mightstone
+
+  # implementation: motor
+  # uri: mongodb:///mongodsn for mongo database
+
+http:
+  cache:
+    directory: /tmp/my-stone/cache