diff --git a/README.md b/README.md index 5e6910e6..a3f38ff0 100644 --- a/README.md +++ b/README.md @@ -1,28 +1,15 @@ # Pyot [![MIT Licensed](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/paaksing/pyot/blob/master/LICENSE) -> ## About the documentation -> The documentation is separated into different pages at the top navbar. -> - **_Core_** section documents the core modules, objects and settings of Pyot. -> - **_Pipeline_** section documents the Low level API of Pyot's Pipeline objects. -> - **_Models_** section documents the objects APIs for each available model. -> - **_Stores_** section documents the available Stores configurable to the pipeline. -> - **_Limiters_** section documents the available Rate Limiters for the RiotAPI Store. -> - **_Utils_** section documents the available helper functions and objects of Pyot. -> - **_Developers_** section has contributing guidelines and wanted features. -> -> Portal: [Pyot Documentations](https://paaksing.github.io/Pyot/) - -Pyot is a Python Framework for the Riot Games API, including League of Legends, Teamfight Tactics, Legends of Runeterra and Valorant. It specializes at doing task in async environment to get the expected result faster than synchronous code. Pyot is highly inspired by [Cassiopeia](https://github.com/meraki-analytics/cassiopeia), you will notice that it has similar approach and structure. +Pyot is a Python Framework for the Riot Games API, including League of Legends, Teamfight Tactics, Legends of Runeterra and Valorant that encourages rapid development and clean, pragmatic design. It specializes at doing task in async environment to get the expected result faster than synchronous code. Thanks for checking it out. -> #### WARNING -> For all users that has Pyot version v1.1.3 or lower please update to v1.1.4 or higher which contains potential fixes to rate limiters. +Pyot is highly inspired by [Cassiopeia](https://github.com/meraki-analytics/cassiopeia), you will notice that both has similar internal workings. ## Features -Read this entirely to get a better idea of what is Pyot possible at. +Features that Pyot has and can provide to your development. -- **_AsyncIO Based_**: No more waiting forever, concurrent calls and jobs made faster, highly configurable settings and wide range of tools to speed you right now. +- **_AsyncIO Based_**: No more waiting forever, concurrent calls and jobs made faster, highly configurable settings and wide range of tools to speed all your I/O tasks. - **_Synchronous Compatible_**: An adapted version of Pyot that runs on synchronous environment, **Pyot will expose part of its API synchronously in its secondary module called Syot**. - **_Django Support_**: Full support for Django Caches Framework and its new 3.1 async Views, just add `pyot` to the installed apps and point your setting modules on your `settings.py` file. - **_Community Projects Integrated_**: Take a step to dump the late and poor updated DDragon, we going beta testing directly using Cdragon and Meraki, BangingHeads' DDragon replacement is also coming soon. @@ -32,183 +19,34 @@ Read this entirely to get a better idea of what is Pyot possible at. - **_Perfect Rate Limiter_**: Pyot Rate Limiter is production tested in all asynchronous, multithreaded and even multiprocessed environments, rate limiters for perfectionists. - **_User Friendly Docs_**: Meet a friendly docs that "should" be easier to read and understand. -## Requirements - -- A computer/laptop with electricity and internet connection. -- Know what is and how to code in Python. -- Ability to read the docs. -- Python version >= 3.7. -- Django version >= 3.0 if used. - -## Installation - -```python -pip install pyot -``` - -## Quick Start - -Activate the Pyot Settings for the model before entering main program, or on the `__init__.py` of your working module. - -```python -from pyot.core import Settings -import os - -Settings( - MODEL = "LOL", - DEFAULT_PLATFORM = "NA1", - DEFAULT_REGION = "AMERICAS", - DEFAULT_LOCALE= "EN_US", - PIPELINE = [ - {"BACKEND": "pyot.stores.Omnistone"}, - {"BACKEND": "pyot.stores.MerakiCDN"}, - {"BACKEND": "pyot.stores.CDragon"}, - { - "BACKEND": "pyot.stores.RiotAPI", - "API_KEY": os.environ["RIOT_API_KEY"], # API KEY - } - ] -).activate() # <- DON'T FORGET TO ACTIVATE THE SETTINGS -``` - -Pyot Settings should be **_activated_** on your main module's `__init__.py` or before your script `main()` entry point. -```python -├─ foo -│ ├─ __init__.py # <---- HERE MOSTLY -│ ├─ __main__.py # <---- OR ANYWHERE BEFORE CALLING `main()` -│ └─ bar.py -# ... -``` - -> This pipeline settings is only specific to League of Legends Model, for example, TFT doesn't have support of the MerakiCDN. - -Now in your main file or module. - -```python -from pyot.models import lol -from pyot.utils import loop_run - -async def main(): - summoner = await lol.Summoner(name="Morimorph", platform="NA1").get() - print(summoner.level) - -loop_run(main()) -``` - -> There is an [issue](https://github.com/aio-libs/aiohttp/issues/4324) on aiohttp related to a `ProactorConnector` Error when used with `asyncio.run()` on Windows (it appears to be closed but more related issue surged because of this), `loop_run()` is the same as `asyncio.get_event_loop().run_until_complete()` imported from the utils module of pyot. - -# Django +## About the Documentation -Plugging Pyot into Django is really easy. +All documentation is in the "docs" directory and online at https://paaksing.github.io/Pyot/. If you're just getting started, here's how we recommend you read the docs: -> #### DEPRECATED -> Since v1.1.0: The module `djot` for Django has been removed, now `pyot` can be installed natively. - -## Installation - -Create a file (the example will use `pipelines.py`) under any of the Django modules (either under an app folder or project folder): - -This example has `test` as the project directory and `pipelines.py` as the module. Inside the file add up the needed Pyot Settings. The below example settings is AN EXAMPLE, you can customize the Settings for your needs. Don't forget to activate the settings. - -```python -#test/pipelines.py - -from pyot.core import Settings -import os - -Settings( - MODEL = "LOL", - DEFAULT_PLATFORM = "NA1", - DEFAULT_REGION = "AMERICAS", - DEFAULT_LOCALE= "EN_US", - PIPELINE = [ - {"BACKEND": "pyot.stores.Omnistone"}, - {"BACKEND": "pyot.stores.MerakiCDN"}, - {"BACKEND": "pyot.stores.CDragon"}, - { - "BACKEND": "pyot.stores.RiotAPI", - "API_KEY": os.environ["RIOT_API_KEY"], - } - ] -).activate() -``` -Then in your projects `settings.py` file, add `pyot` to the `INSTALLED_APPS`. -```python -INSTALLED_APPS = [ - 'django.contrib.admin', - 'django.contrib.auth', - 'django.contrib.contenttypes', - 'django.contrib.sessions', - 'django.contrib.messages', - 'django.contrib.staticfiles', - 'pyot', -] -``` -In the same `settings.py` file, add the file path to a reserved variable for Pyot called `PYOT_SETTINGS`. -```python -# Supposing the pyot settings file is at: test/pipelines.py - -PYOT_SETTINGS = [ - 'test.pipelines' -] -``` -You can define multiple settings in different files if you want to keep 1 setting per app (supposing you have 1 app per game model). - -# Syot - -Syot is a back ported version of Pyot to synchronous code (rarely a case not going async), this might be an option for those who don't want to go async or wants flexibility by using both async and sync code at the same time, which is in some case for Django views. - ->You still need to activate the Settings for Syot to work. - ->Syot and Pyot **_shares the same pipeline_** per each model so you can use both environment together without problem of any. They won't have any conflict unless you try to activate the same Settings twice both in Syot and Pyot. - -Below documentation only applies to Syot. -The rest of the API please refer to Pyot documentation by replacing `pyot` with `syot` instead, awaitables needs to be executed with `loop_run()`. - -## Similarities -1. All Pyot Object's methods that are not marked with "awaitable" are usable in Syot, that includes Pyot Object's `dict()`, `json()` and others not mentioned. -2. All the models API are available on Syot, with some minor changes listed below. - -## Differences -1. Lose the advantage of cooperative tasks and high concurrency to speed up the calls. -2. The Pyot Pipeline Low Level API is not available in synchronous environment, you would need to do `loop_run()` for every single pipeline coroutine. -3. The Pyot Gatherer is also not supported here, because it is a feature only for asynchrounous environment. -4. Instead of `from pyot` do `from syot` to import the synchronous version of Pyot. -5. You no longer need to `await` the `get()` methods on the Objects, and `get()` is now "chainable", meaning you can chain attributes and other methods right after `get()`. +> The documentation is separated into different pages at the top navbar. +> - **_Core_** section documents the core modules, objects and settings of Pyot. +> - **_Pipeline_** section documents the Low level API of Pyot's Pipeline objects. +> - **_Models_** section documents the objects APIs for each available model. +> - **_Stores_** section documents the available Stores configurable to the pipeline. +> - **_Limiters_** section documents the available Rate Limiters for the RiotAPI Store. +> - **_Utils_** section documents the available helper functions and objects of Pyot. +> - **_Developers_** section has contributing guidelines and wanted features. +> +> Portal: https://paaksing.github.io/Pyot/ -## Example Usage -Activate the settings before you script entry point or module `__init__.py` -```python -from syot.core import Settings -import os +1. First, read **Core > Introduction > Installation Guide** for instructions on installing Pyot. +2. Next, follow the quick start guide in **Core > Introduction > Quick Start Guide** for creating and running your first Pyot project. +3. Then you should get to know the types of objects that Pyot works with in **Core > Cores > Objects**. +4. Now give yourself an idea of what models we have and what objects we work in **Models** +5. You'll probably want to read through the topical context managers for achieving concurrency in **Core > Cores > Gatherer** and **Core > Cores > Queue**. +6. From there you can jump back to manipulating the settings by reading **Core > Cores > Settings** and get to know all the available pipeline stores in Pyot at **Stores**. -Settings( - MODEL = "LOL", - DEFAULT_PLATFORM = "NA1", - DEFAULT_REGION = "AMERICAS", - DEFAULT_LOCALE= "EN_US", - PIPELINE = [ - {"BACKEND": "pyot.stores.Omnistone"}, - {"BACKEND": "pyot.stores.MerakiCDN"}, - {"BACKEND": "pyot.stores.CDragon"}, - { - "BACKEND": "pyot.stores.RiotAPI", - "API_KEY": os.environ["RIOT_API_KEY"], # API KEY - } - ] -).activate() # <- DON'T FORGET TO ACTIVATE THE SETTINGS -``` -Example of Syot code -```python -from syot.models import lol +Docs are updated rigorously. If you find any problems in the docs, or think they should be clarified in any way, please take 30 seconds to open an issue in this repository. -summoner = lol.Summoner(name="Morimorph", platform="NA1").get() -print(summoner.level) +## To contribute to Pyot -#OR using method chains: -print(lol.Summoner(name="Morimorph", platform="NA1").get().level) -``` +Contributions are welcome! If you have idea or opinions on how things can be improved, don’t hesitate to let us know by posting an issue on GitHub or @ing me on the Riot API Discord channel. And we always want to hear from our users, even (especially) if it’s just letting us know how you are using Pyot. -# Contributing +Check out https://paaksing.github.io/Pyot/devs/ for information about getting involved. -Contributions are welcome! If you have idea or opinions on how things can be improved, don’t hesitate to let us know by posting an issue on GitHub or @ing me on the Riot API Discord channel. And we always want to hear from our users, even (especially) if it’s just letting us know how you are using Pyot. +Finally thanks for Django docs, I literally copied their doc format and changed the names. Yikes diff --git a/README_pre_2.md b/README_pre_2.md new file mode 100644 index 00000000..5e6910e6 --- /dev/null +++ b/README_pre_2.md @@ -0,0 +1,214 @@ +# Pyot +[![MIT Licensed](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/paaksing/pyot/blob/master/LICENSE) + +> ## About the documentation +> The documentation is separated into different pages at the top navbar. +> - **_Core_** section documents the core modules, objects and settings of Pyot. +> - **_Pipeline_** section documents the Low level API of Pyot's Pipeline objects. +> - **_Models_** section documents the objects APIs for each available model. +> - **_Stores_** section documents the available Stores configurable to the pipeline. +> - **_Limiters_** section documents the available Rate Limiters for the RiotAPI Store. +> - **_Utils_** section documents the available helper functions and objects of Pyot. +> - **_Developers_** section has contributing guidelines and wanted features. +> +> Portal: [Pyot Documentations](https://paaksing.github.io/Pyot/) + +Pyot is a Python Framework for the Riot Games API, including League of Legends, Teamfight Tactics, Legends of Runeterra and Valorant. It specializes at doing task in async environment to get the expected result faster than synchronous code. Pyot is highly inspired by [Cassiopeia](https://github.com/meraki-analytics/cassiopeia), you will notice that it has similar approach and structure. + +> #### WARNING +> For all users that has Pyot version v1.1.3 or lower please update to v1.1.4 or higher which contains potential fixes to rate limiters. + +## Features + +Read this entirely to get a better idea of what is Pyot possible at. + +- **_AsyncIO Based_**: No more waiting forever, concurrent calls and jobs made faster, highly configurable settings and wide range of tools to speed you right now. +- **_Synchronous Compatible_**: An adapted version of Pyot that runs on synchronous environment, **Pyot will expose part of its API synchronously in its secondary module called Syot**. +- **_Django Support_**: Full support for Django Caches Framework and its new 3.1 async Views, just add `pyot` to the installed apps and point your setting modules on your `settings.py` file. +- **_Community Projects Integrated_**: Take a step to dump the late and poor updated DDragon, we going beta testing directly using Cdragon and Meraki, BangingHeads' DDragon replacement is also coming soon. +- **_Caches Integrated_**: A wide range of Caches Stores is available right out of the box, we currently have Omnistone(Runtime), RedisCache(RAM), DiskCache(Disk) and MongoDB(NoSQL). +- **_Multiple Models_**: Available models are League of Legends, Teamfight Tactics, Legends of Runeterra and Valorant. +- **_Autocompletion Included_**: Forget the standard dictionary keys, triple your code efficiency now with autocompletion enabled. +- **_Perfect Rate Limiter_**: Pyot Rate Limiter is production tested in all asynchronous, multithreaded and even multiprocessed environments, rate limiters for perfectionists. +- **_User Friendly Docs_**: Meet a friendly docs that "should" be easier to read and understand. + +## Requirements + +- A computer/laptop with electricity and internet connection. +- Know what is and how to code in Python. +- Ability to read the docs. +- Python version >= 3.7. +- Django version >= 3.0 if used. + +## Installation + +```python +pip install pyot +``` + +## Quick Start + +Activate the Pyot Settings for the model before entering main program, or on the `__init__.py` of your working module. + +```python +from pyot.core import Settings +import os + +Settings( + MODEL = "LOL", + DEFAULT_PLATFORM = "NA1", + DEFAULT_REGION = "AMERICAS", + DEFAULT_LOCALE= "EN_US", + PIPELINE = [ + {"BACKEND": "pyot.stores.Omnistone"}, + {"BACKEND": "pyot.stores.MerakiCDN"}, + {"BACKEND": "pyot.stores.CDragon"}, + { + "BACKEND": "pyot.stores.RiotAPI", + "API_KEY": os.environ["RIOT_API_KEY"], # API KEY + } + ] +).activate() # <- DON'T FORGET TO ACTIVATE THE SETTINGS +``` + +Pyot Settings should be **_activated_** on your main module's `__init__.py` or before your script `main()` entry point. +```python +├─ foo +│ ├─ __init__.py # <---- HERE MOSTLY +│ ├─ __main__.py # <---- OR ANYWHERE BEFORE CALLING `main()` +│ └─ bar.py +# ... +``` + +> This pipeline settings is only specific to League of Legends Model, for example, TFT doesn't have support of the MerakiCDN. + +Now in your main file or module. + +```python +from pyot.models import lol +from pyot.utils import loop_run + +async def main(): + summoner = await lol.Summoner(name="Morimorph", platform="NA1").get() + print(summoner.level) + +loop_run(main()) +``` + +> There is an [issue](https://github.com/aio-libs/aiohttp/issues/4324) on aiohttp related to a `ProactorConnector` Error when used with `asyncio.run()` on Windows (it appears to be closed but more related issue surged because of this), `loop_run()` is the same as `asyncio.get_event_loop().run_until_complete()` imported from the utils module of pyot. + +# Django + +Plugging Pyot into Django is really easy. + +> #### DEPRECATED +> Since v1.1.0: The module `djot` for Django has been removed, now `pyot` can be installed natively. + +## Installation + +Create a file (the example will use `pipelines.py`) under any of the Django modules (either under an app folder or project folder): + +This example has `test` as the project directory and `pipelines.py` as the module. Inside the file add up the needed Pyot Settings. The below example settings is AN EXAMPLE, you can customize the Settings for your needs. Don't forget to activate the settings. + +```python +#test/pipelines.py + +from pyot.core import Settings +import os + +Settings( + MODEL = "LOL", + DEFAULT_PLATFORM = "NA1", + DEFAULT_REGION = "AMERICAS", + DEFAULT_LOCALE= "EN_US", + PIPELINE = [ + {"BACKEND": "pyot.stores.Omnistone"}, + {"BACKEND": "pyot.stores.MerakiCDN"}, + {"BACKEND": "pyot.stores.CDragon"}, + { + "BACKEND": "pyot.stores.RiotAPI", + "API_KEY": os.environ["RIOT_API_KEY"], + } + ] +).activate() +``` +Then in your projects `settings.py` file, add `pyot` to the `INSTALLED_APPS`. +```python +INSTALLED_APPS = [ + 'django.contrib.admin', + 'django.contrib.auth', + 'django.contrib.contenttypes', + 'django.contrib.sessions', + 'django.contrib.messages', + 'django.contrib.staticfiles', + 'pyot', +] +``` +In the same `settings.py` file, add the file path to a reserved variable for Pyot called `PYOT_SETTINGS`. +```python +# Supposing the pyot settings file is at: test/pipelines.py + +PYOT_SETTINGS = [ + 'test.pipelines' +] +``` +You can define multiple settings in different files if you want to keep 1 setting per app (supposing you have 1 app per game model). + +# Syot + +Syot is a back ported version of Pyot to synchronous code (rarely a case not going async), this might be an option for those who don't want to go async or wants flexibility by using both async and sync code at the same time, which is in some case for Django views. + +>You still need to activate the Settings for Syot to work. + +>Syot and Pyot **_shares the same pipeline_** per each model so you can use both environment together without problem of any. They won't have any conflict unless you try to activate the same Settings twice both in Syot and Pyot. + +Below documentation only applies to Syot. +The rest of the API please refer to Pyot documentation by replacing `pyot` with `syot` instead, awaitables needs to be executed with `loop_run()`. + +## Similarities +1. All Pyot Object's methods that are not marked with "awaitable" are usable in Syot, that includes Pyot Object's `dict()`, `json()` and others not mentioned. +2. All the models API are available on Syot, with some minor changes listed below. + +## Differences +1. Lose the advantage of cooperative tasks and high concurrency to speed up the calls. +2. The Pyot Pipeline Low Level API is not available in synchronous environment, you would need to do `loop_run()` for every single pipeline coroutine. +3. The Pyot Gatherer is also not supported here, because it is a feature only for asynchrounous environment. +4. Instead of `from pyot` do `from syot` to import the synchronous version of Pyot. +5. You no longer need to `await` the `get()` methods on the Objects, and `get()` is now "chainable", meaning you can chain attributes and other methods right after `get()`. + +## Example Usage +Activate the settings before you script entry point or module `__init__.py` +```python +from syot.core import Settings +import os + +Settings( + MODEL = "LOL", + DEFAULT_PLATFORM = "NA1", + DEFAULT_REGION = "AMERICAS", + DEFAULT_LOCALE= "EN_US", + PIPELINE = [ + {"BACKEND": "pyot.stores.Omnistone"}, + {"BACKEND": "pyot.stores.MerakiCDN"}, + {"BACKEND": "pyot.stores.CDragon"}, + { + "BACKEND": "pyot.stores.RiotAPI", + "API_KEY": os.environ["RIOT_API_KEY"], # API KEY + } + ] +).activate() # <- DON'T FORGET TO ACTIVATE THE SETTINGS +``` +Example of Syot code +```python +from syot.models import lol + +summoner = lol.Summoner(name="Morimorph", platform="NA1").get() +print(summoner.level) + +#OR using method chains: +print(lol.Summoner(name="Morimorph", platform="NA1").get().level) +``` + +# Contributing + +Contributions are welcome! If you have idea or opinions on how things can be improved, don’t hesitate to let us know by posting an issue on GitHub or @ing me on the Riot API Discord channel. And we always want to hear from our users, even (especially) if it’s just letting us know how you are using Pyot. diff --git a/docs/src/.vuepress/config.js b/docs/src/.vuepress/config.js index e5cac71d..d79e8d6e 100644 --- a/docs/src/.vuepress/config.js +++ b/docs/src/.vuepress/config.js @@ -82,12 +82,21 @@ module.exports = { collapsable: false, children: [ '', - 'syot', + 'installation', + 'startup', + ] + }, + { + title: 'Integrations', + collapsable: false, + children: [ 'django', + 'celery', + 'syot', ] }, { - title: 'Core', + title: 'Cores', collapsable: false, sidebarDepth: 2, children: [ diff --git a/docs/src/core/apiobjects.md b/docs/src/core/apiobjects.md index 065cd687..70a5b547 100644 --- a/docs/src/core/apiobjects.md +++ b/docs/src/core/apiobjects.md @@ -11,10 +11,11 @@ This is main type of objects that developers works with. Below is a list of gene > ### `__init__(**kwargs)` > Creates an instance of the Pyot Core Object. Parameters varies per API. -> ### `get(sid: str = None, pipeline: str = None)` +> ### `get(sid: str = None, pipeline: str = None, ptr_cache: PtrCache = None)` > Awaitable that executes `get` request to the pipeline, finds the requested data, returns it and sinks through the pipeline. > - `sid` : Optional, provide the sid identifying the created session on the pipeline to reuse, typically session created by `Queue`. > - `pipeline` : Optional, provide the name identifying the pipeline to execute on, typically only passed when used with objects of the RIOT model. +> - `ptr_cache` : Optional, Intercepts a PtrCache, usage details please refer to [PtrCache](/utils/objects.html#PtrCache). > ### `post(sid: str = None, pipeline: str = None)` > Awaitable that executes `post` request to the pipeline, finds the correct service to execute and return the response if given. Unlike `get()` responses are not sinked through the pipeline. diff --git a/docs/src/core/celery.md b/docs/src/core/celery.md new file mode 100644 index 00000000..32b913d1 --- /dev/null +++ b/docs/src/core/celery.md @@ -0,0 +1,54 @@ +# Celery + +Celery is Task Queue for scheduling tasks and workloads. + +You probably don't need celery for concurrency, since the Pyot itself gives you high concurrency. In most scenario you will need celery for distributing work across processes or machines, which in Python it can become tricky with `multiprocessing` and periodic tasks that can also get tricky with normal cronjobs. + +Example usage: +* Run a task every day at 4 AM. +* Run multiple CPU intense tasks at the same time in a multicore machine +* Run the same task in multiple machines. + +Celery itself supports Django. + +First you need to follow the [celery setup guide](https://docs.celeryproject.org/en/stable/getting-started/introduction.html). + +After that, all you need to do is import the celery instance and decorate it as needed. + +## Asyncio Issues + +Celery does not support async functions, there is 2 solution. + +### Using AsyncToSync decorator + +```shell +pip install asgiref +``` +Wrap the tasks + +```python +from asgiref.sync import AsyncToSync +from .celery import app + +@app.task +@AsyncToSync +async def mytask(): + # ... +``` + +### Calling asyncio.run + +This one involves in creating a proxy synchronous function, not so clean. You can also create a decorator that functions similarly to `AsyncToSync`. + +```python +import asyncio +from .celery import app + +async def mytask(): + # ... + +@app.task +def my_proxy_task(): + asyncio.run(mytask()) +``` + diff --git a/docs/src/core/django.md b/docs/src/core/django.md index a2e4629c..aa8187cc 100644 --- a/docs/src/core/django.md +++ b/docs/src/core/django.md @@ -1,19 +1,15 @@ # Django -Plugging Pyot into Django is really easy. - -:::danger DEPRECATED -Since v1.1.0: The module `djot` for Django has been removed, now `pyot` can be installed natively. -::: +Integrating Pyot into Django is easy. ## Installation Create a file (the example will use `pipelines.py`) under any of the Django modules (either under an app folder or project folder): -This example has `test` as the project directory and `pipelines.py` as the module. Inside the file add up the needed Pyot Settings. The below example settings is AN EXAMPLE, you can customize the Settings for your needs. Don't forget to activate the settings. +This example has `mysite` as the project directory and `pipelines.py` as the module. Inside the file add up the needed Pyot Settings. The below example settings is AN EXAMPLE, you can customize the Settings for your needs. Don't forget to activate the settings. ```python{21} -#test/pipelines.py +#mysite/pipelines.py from pyot.core import Settings import os @@ -48,10 +44,8 @@ INSTALLED_APPS = [ ``` In the same `settings.py` file, add the file path to a reserved variable for Pyot called `PYOT_SETTINGS`. ```python -# Supposing the pyot settings file is at: test/pipelines.py +# Supposing the pyot settings file is at: mysite/pipelines.py -PYOT_SETTINGS = [ - 'test.pipelines' -] +PYOT_SETTINGS = ['mysite.pipelines'] ``` You can define multiple settings in different files if you want to keep 1 setting per app (supposing you have 1 app per game model). diff --git a/docs/src/core/gatherer.md b/docs/src/core/gatherer.md index a595ba46..83fe7c84 100644 --- a/docs/src/core/gatherer.md +++ b/docs/src/core/gatherer.md @@ -2,14 +2,16 @@ This is an object to gather data by chunks, creates a session and store this session under each of the pipelines `sessions` dict with an `uuid4()` key to give the highest uniqueness possible, this session id is added to each of the objects in the `statements` provided. After finishing the execution of all the statements, it then proceed to delete the session from the pipeline. -:::warning -All of the `statements` provided needs to be an instance of the Pyot Core object and **_`get()` must NOT be appended to the instance, as `get()` is "unchainable" so Pyot Gatherer has no way of calling `get(sid=...)` again_**. It will raise a `RuntimeError` if this happens. +:::tip ABOUT +This manager is only recommended for working with small amount of Pyot Core Objects for short time bursts. If you need to execute custom tasks written by you or working with large amount of objects, you should go with [Queue](queue.html) instead. ::: -:::tip INFO + +:::warning INFO +All of the `statements` provided needs to be an instance of the Pyot Core object and **_`get()` must NOT be appended to the instance, as `get()` is "unchainable" so Pyot Gatherer has no way of calling `get(sid=...)` again_**. It will raise a `RuntimeError` if this happens. + `query()` and other methods (that are not coroutines) that returns `self` **_are safe and can be appended to the `statement`_**. Pyot Gatherer will automatically append `get()` to the instance after setting the session id. -::: -:::tip FUN FACT + Gatherer workers are NOT real workers, that is the size of the chunk to gather. ::: :::danger READ ME @@ -20,12 +22,7 @@ async with Gatherer() as gatherer: gatherer.statements = matches await gatherer.gather() # <-- This will load aprox. 10 GB to memory ``` -The why of this is because `matches` is a mutable object, so it will be passed to the gatherer by reference, it will endup filling up the 30k matches on that list, a workaround would be to load your matches in a loop scope so Python can garbage collect it, or better use the utils `FrozenGenerator` plus the other gathering tool `Queue`. -::: - -## Pyot Settings Reference -::: danger DEPRECATED -Pyot Settings for Gatherer has been removed since v1.1.0, due to confusing settings. +The why of this is because `matches` is a mutable object, so it will be passed to the gatherer by reference, it will endup filling up the 30k matches on that list, a workaround would be to load your matches in a loop scope so Python can garbage collect it, or better use the utils `FrozenGenerator` plus the other gathering tool `Queue`. ::: ## Pyot Gatherer API @@ -44,9 +41,6 @@ This object is preferably used as a context manager because it will clean up the > ### `__init__(workers: int = 25, log_level: int = 10, cancel_on_raise: bool = False)` > Creates an instance of Gatherer with the respective params, these params are set when Pyot Settings was set if specified the `GATHERER` param, you can also override partial settings at runtime by passing the params on instance creation: > - `workers` -> `int`: Maximum number of concurrent connections and tasks allowed for this Gatherer. Increasing the number of workers may increase or decrease performance. Defaults to 25. ->:::danger DEPRECATED -> Since v1.1.0: The `session_class` param is removed, due to useless param. ->::: > - `log_level` -> `bool`: Set the log level for the Gatherer (does not affect pipeline logs). Defaults to 10 (DEBUG level). > - `cancel_on_raise` -> `bool`: Cancel all remaining tasks if one raises exception. Defaults to `False`. @@ -76,7 +70,7 @@ from typing import List async def pull_leagues(): league = await lol.ChallengerLeague(queue="RANKED_SOLO_5x5", platform="NA1").get() async with Gatherer() as gatherer: # type: pyot.Gatherer - gatherer.statements = [entry.summoner for entry in league.entries[:62]] + gatherer.statements = [entry.summoner for entry in league.entries] responses = await gatherer.gather() # type: List[pyot.lol.Summoner] for r in responses: print(r.profile_icon_id) diff --git a/docs/src/core/index.md b/docs/src/core/index.md index dd28b870..413480cb 100644 --- a/docs/src/core/index.md +++ b/docs/src/core/index.md @@ -1,28 +1,15 @@ # Pyot -##### [](https://github.com/paaksing/pyot/blob/master/LICENSE) +##### [](https://github.com/paaksing/pyot/blob/master/LICENSE) -:::tip ABOUT THIS DOCUMENTATION -The documentation is separated into different pages at the top navbar. -- **_Core_** section documents the core modules, objects and settings of Pyot. -- **_Pipeline_** section documents the Low level API of Pyot's Pipeline objects. -- **_Models_** section documents the objects APIs for each available model. -- **_Stores_** section documents the available Stores configurable to the pipeline. -- **_Limiters_** section documents the available Rate Limiters for the RiotAPI Store. -- **_Utils_** section documents the available helper functions and objects of Pyot. -- **_Developers_** section has contributing guidelines and wanted features. -::: +Pyot is a Python Framework for the Riot Games API, including League of Legends, Teamfight Tactics, Legends of Runeterra and Valorant that encourages rapid development and clean, pragmatic design. It specializes at doing task in async environment to get the expected result faster than synchronous code. Thanks for checking it out. -Pyot is a Python Framework for the Riot Games API, including League of Legends, Teamfight Tactics, Legends of Runeterra and Valorant. It specializes at doing task in async environment to get the expected result faster than synchronous code. Pyot is highly inspired by [Cassiopeia](https://github.com/meraki-analytics/cassiopeia), you will notice that it has similar approach and structure. - -::: danger WARNING -For all users that has Pyot version v1.1.3 or lower please update to v1.1.4 or higher which contains potential fixes to rate limiters. -::: +Pyot is highly inspired by [Cassiopeia](https://github.com/meraki-analytics/cassiopeia), you will notice that both has similar internal workings. ## Features -Read this entirely to get a better idea of what is Pyot possible at. +Features that Pyot has and can provide to your development. -- **_AsyncIO Based_**: No more waiting forever, concurrent calls and jobs made faster, highly configurable settings and wide range of tools to speed you right now. +- **_AsyncIO Based_**: No more waiting forever, concurrent calls and jobs made faster, highly configurable settings and wide range of tools to speed all your I/O tasks. - **_Synchronous Compatible_**: An adapted version of Pyot that runs on synchronous environment, **Pyot will expose part of its API synchronously in its secondary module called [Syot](syot.html)** . - **_Django Support_**: Full support for Django Caches Framework and its new 3.1 async Views, just add `pyot` to the installed apps and point your setting modules on your `settings.py` file. [More details](django.html). - **_Community Projects Integrated_**: Take a step to dump the late and poor updated DDragon, we going beta testing directly using Cdragon and Meraki, BangingHeads' DDragon replacement is also coming soon. @@ -32,78 +19,32 @@ Read this entirely to get a better idea of what is Pyot possible at. - **_Perfect Rate Limiter_**: Pyot Rate Limiter is production tested in all asynchronous, multithreaded and even multiprocessed environments, rate limiters for perfectionists. - **_User Friendly Docs_**: Meet a friendly docs that "should" be easier to read and understand. -## Requirements - -- A computer/laptop with electricity and internet connection. -- Know what is and how to code in Python. -- Ability to read the docs. -- Python version >= 3.7. -- Django version >= 3.0 if used. - -## Installation - -```python -pip install pyot -``` - -## Quick Start - -:::tip -For Django Setup, please refer to Django section on the sidebar. -::: +## About the Documentation -Activate the Pyot Settings for the model before entering main program, or on the `__init__.py` of your working module. +All documentation is in the "docs" directory and online at https://paaksing.github.io/Pyot/. If you're just getting started, here's how we recommend you read the docs: -```python{15,18} -from pyot.core import Settings -import os +> The documentation is separated into different pages at the top navbar. +> - **_Core_** section documents the core modules, objects and settings of Pyot. +> - **_Pipeline_** section documents the Low level API of Pyot's Pipeline objects. +> - **_Models_** section documents the objects APIs for each available model. +> - **_Stores_** section documents the available Stores configurable to the pipeline. +> - **_Limiters_** section documents the available Rate Limiters for the RiotAPI Store. +> - **_Utils_** section documents the available helper functions and objects of Pyot. +> - **_Developers_** section has contributing guidelines and wanted features. -Settings( - MODEL = "LOL", - DEFAULT_PLATFORM = "NA1", - DEFAULT_REGION = "AMERICAS", - DEFAULT_LOCALE= "EN_US", - PIPELINE = [ - {"BACKEND": "pyot.stores.Omnistone"}, - {"BACKEND": "pyot.stores.MerakiCDN"}, - {"BACKEND": "pyot.stores.CDragon"}, - { - "BACKEND": "pyot.stores.RiotAPI", - "API_KEY": os.environ["RIOT_API_KEY"], # API KEY - } - ] -).activate() # <- DON'T FORGET TO ACTIVATE THE SETTINGS -``` +1. First, read **Core > Introduction > Installation Guide** for instructions on installing Pyot. +2. Next, follow the quick start guide in **Core > Introduction > Quick Start Guide** for creating and running your first Pyot project. +3. Then you should get to know the types of objects that Pyot works with in **Core > Cores > Objects**. +4. Now give yourself an idea of what models we have and what objects we work in **Models** +5. You'll probably want to read through the topical context managers for achieving concurrency in **Core > Cores > Gatherer** and **Core > Cores > Queue**. +6. From there you can jump back to manipulating the settings by reading **Core > Cores > Settings** and get to know all the available pipeline stores in Pyot at **Stores**. -Pyot Settings should be **_activated_** on your main module's `__init__.py` or before your script `main()` entry point. -```python -├─ foo -│ ├─ __init__.py # <---- HERE MOSTLY -│ ├─ __main__.py # <---- OR ANYWHERE BEFORE CALLING `main()` -│ └─ bar.py -# ... -``` +Docs are updated rigorously. If you find any problems in the docs, or think they should be clarified in any way, please take 30 seconds to open an issue in this repository. -:::tip NOTE -This pipeline settings is only specific to League of Legends Model, for example, TFT doesn't have support of the MerakiCDN. -::: +## To contribute to Pyot -Now in your main file or module. - -```python{5} -from pyot.models import lol -from pyot.utils import loop_run - -async def main(): - summoner = await lol.Summoner(name="Morimorph", platform="NA1").get() - print(summoner.level) - -loop_run(main()) -``` -::: tip INFO -There is an [issue](https://github.com/aio-libs/aiohttp/issues/4324) on aiohttp related to a `ProactorConnector` Error when used with `asyncio.run()` on Windows (it appears to be closed but more related issue surged because of this), `loop_run()` is the same as `asyncio.get_event_loop().run_until_complete()` imported from the utils module of pyot. -::: +Contributions are welcome! If you have idea or opinions on how things can be improved, don’t hesitate to let us know by posting an issue on GitHub or @ing me on the Riot API Discord channel. And we always want to hear from our users, even (especially) if it’s just letting us know how you are using Pyot. -## Contributing +Check out https://paaksing.github.io/Pyot/devs/ for information about getting involved. -Contributions are welcome! If you have idea or opinions on how things can be improved, don’t hesitate to let us know by posting an issue on GitHub or @ing me on the Riot API Discord channel. And we always want to hear from our users, even (especially) if it’s just letting us know how you are using Pyot. +Finally thanks for Django docs, I literally copied their doc format and changed the names. Yikes diff --git a/docs/src/core/installation.md b/docs/src/core/installation.md new file mode 100644 index 00000000..9a695ded --- /dev/null +++ b/docs/src/core/installation.md @@ -0,0 +1,45 @@ +# Installation Guide + +Before you can use Pyot, you’ll need to get it installed. This guide will guide you to a minimal installation that’ll work while you walk through the introduction. + +## Install Python + +Being a Python framework, Pyot requires Python v3.7 or higher. + +Get the latest version of Python at https://www.python.org/downloads/ or with your operating system’s package manager. + +You can verify that Python is installed by typing `python` from your shell; you should see something like: + +```shell +Python 3.x.y +[GCC 4.x] on linux +Type "help", "copyright", "credits" or "license" for more information. +>>> +``` + +## Install Pyot Code + +Installing an official release with pip +This is the recommended way to install Pyot. + +Install [pip](https://pip.pypa.io/). The easiest is to use the [standalone pip installer](https://pip.pypa.io/en/latest/installing/#installing-with-get-pip-py). If your distribution already has pip installed, you might need to update it if it’s outdated. If it’s outdated, you’ll know because installation won’t work. + +Take a look at [venv](https://docs.python.org/3/tutorial/venv.html). This tool provides isolated Python environments, which are more practical than installing packages systemwide. It also allows installing packages without administrator privileges. + +* Windows +```shell +python -m venv venv +venv\\scripts\\activate +``` + +* Unix +```shell +python -m venv venv +source venv/bin/activate +``` + +After you’ve created and activated a virtual environment, enter the command: + +```shell +pip install pyot +``` diff --git a/docs/src/core/startup.md b/docs/src/core/startup.md new file mode 100644 index 00000000..f8253670 --- /dev/null +++ b/docs/src/core/startup.md @@ -0,0 +1,87 @@ +# Quick Start Guide + +Build your first pyot project. + +::: warning Before starting this guide +* This guide only applies to pyot v2.0.0 or higher. +* For integrating with Django, there is a total different [setup](django.html) +::: + +## Create a new project + +If this is your first time using Pyot, you’ll have to take care of some initial setup. Namely, you’ll need to change the environments that establishes a Pyot project – a collection of settings for an instance of Pyot, including API key setup, store specific options and pipeline-specific settings. + +From the command line, cd into a directory where you’d like to store your code, then run the following command: + +```shell +pyot startproject myproject +``` +This will create a **myproject** directory in your current directory. + +:::warning +You’ll need to avoid naming projects after built-in Python or Pyot components. In particular, this means you should avoid using names like django (which will conflict with Pyot itself) or test (which conflicts with a built-in Python package). +::: + +```shell +myproject/ + __init__.py + __main__.py + manage.py + settings.py + tasks.py +``` + +These files are: +* **__init__.py**: Marks the directory as module, imports the Pyot settings from **settings.py** and others. +* **__main__.py**: Marks the module as execurable, entry point of the module, defaults to executing the task manager in **manage.py**. +* **manage.py**: Managers for this Pyot project, you can change this file to your wanted behavior. +* **settings.py**: Settings/Configuration for this Pyot project, [Pyot settings](settings.html) will tell you how settings works. +* **tasks.py**: Tasks of this Pyot project, all callables in this file will be accessible by managers. + +## Running your project for the first time + +There is a default task added by `startproject` command + +First, please that your API key is set, Pyot will default to get your environment variable called `RIOT_API_KEY`, you can change the name of the variable to match yours if needed, ***hardcoding your API key is not recommended*** because it will risk API key leakage. + +```python +# settings.py + # ... + 'API_KEY': os.environ['RIOT_API_KEY'] +``` + +Now everything is set up, run the following command: + +```shell +python -m myproject summoner_level Morimorph NA1 +``` + +You should get printed in your console: +```shell +Morimorph in NA1 is level +``` + +Congrats! Your first project is all set up. + +## How it works + +We executed above: +```shell +python -m myproject summoner_level Morimorph NA1 +``` + +1. `python -m myproject` tells python to execute `myproject` as a module, which then translates into calling the manager in `__main__.py`. +2. `summoner_level` tells the manager the name of task in `tasks.py` to load and execute. +3. `Morimorph NA1` are the arguments that the callable `summoner_level` takes, it will be unpacked in the same order as given. + +Below is source code for `summoner_level` in `tasks.py`: + +```python +async def summoner_level(name, platform): + '''Get summoner level by name and platform''' + try: + summoner = await lol.Summoner(name=name, platform=platform).get() + print(summoner.name, 'in', summoner.platform.upper(), 'is level', summoner.level) + except NotFound: + print('Summoner not found') +``` diff --git a/docs/src/core/syot.md b/docs/src/core/syot.md index 320253e7..016e2522 100644 --- a/docs/src/core/syot.md +++ b/docs/src/core/syot.md @@ -1,6 +1,6 @@ # Syot -Syot is a back ported version of Pyot to synchronous code (rarely a case not going async), this might be an option for those who don't want to go async or wants flexibility by using both async and sync code at the same time, which is in some case for Django views. +Syot is a back ported version of Pyot to synchronous code (rarely a case not going async), this might be an option for those who don't want to go async or wants less complexity in code. :::tip INFO You still need to activate the Settings for Syot to work. diff --git a/manage.py b/manage.py deleted file mode 100644 index 29acdd52..00000000 --- a/manage.py +++ /dev/null @@ -1,22 +0,0 @@ -#!/usr/bin/env python -"""Django's command-line utility for administrative tasks.""" -import os -import sys - - -def main(): - """Run administrative tasks.""" - os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'djangotest.settings') - try: - from django.core.management import execute_from_command_line - except ImportError as exc: - raise ImportError( - "Couldn't import Django. Are you sure it's installed and " - "available on your PYTHONPATH environment variable? Did you " - "forget to activate a virtual environment?" - ) from exc - execute_from_command_line(sys.argv) - - -if __name__ == '__main__': - main() diff --git a/pyot/core/objects.py b/pyot/core/objects.py index 85508cf7..d37deb0e 100644 --- a/pyot/core/objects.py +++ b/pyot/core/objects.py @@ -1,4 +1,3 @@ -from functools import wraps, partial from typing import Dict, List, Mapping, Any, get_type_hints import pickle import re @@ -18,11 +17,7 @@ class PyotLazyObject: clas: Any def __init__(self, clas, obj, server_type, server): - try: - if clas.__origin__ is list: - self.clas = clas.__args__[0] - except Exception: - self.clas = clas + self.clas = clas self.server_map = [server_type, server] self.obj = obj @@ -85,52 +80,41 @@ def __init__(self, data): # META CLASS UNIQUE MUTABLE OBJECTS self._meta = self.Meta() self._meta.data = data - self._meta.types = typing_cache.get(self.__class__, partial(get_type_hints, self.__class__)) + self._meta.types = typing_cache.get(self.__class__, self._get_types) def __getattribute__(self, name): try: attr = super().__getattribute__(name) - if type(attr) is PyotLazyObject: + if isinstance(attr, PyotLazyObject): obj = attr() setattr(self, name, obj) return obj else: return attr - except (KeyError, AttributeError): - raise AttributeError(f"'{self.__class__.__name__}' object has no attribute '{name}'") + except (KeyError, AttributeError) as e: + raise AttributeError(f"'{self.__class__.__name__}' object has no attribute '{name}'") from e def __getitem__(self, item): return self._meta.data[item] - def _rename(self, data): - # SNAKECASE > RENAME > REMOVE - new_data = {} - mapping = {} - for attr, val in data.items(): - name = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', attr) - newkey = re.sub('([a-z0-9])([A-Z])', r'\1_\2', name).lower() - if newkey in self._meta.removed: - continue - if newkey in self._meta.renamed: - newkey = self._meta.renamed[newkey] - mapping[attr] = newkey - new_data[newkey] = val - - return new_data, mapping - - def _normalize(self, data): - mapping = normalizer_cache.get(self.__class__, dict) - new_data = {} - for attr, val in data.items(): + def _get_types(self): + types = get_type_hints(self.__class__) + for typ, clas in types.items(): try: - new_data[mapping[attr]] = val - except KeyError: - new_data, new_mapping = self._rename(data) - mapping.update(new_mapping) - return new_data - for key in self._meta.removed: - new_data.pop(key, None) - return new_data + if clas.__origin__ is list: + types[typ] = clas.__args__[0] + except Exception: + pass + return types + + def _rename(self, key): + name = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', key) + newkey = re.sub('([a-z0-9])([A-Z])', r'\1_\2', name).lower() + if newkey in self._meta.removed: + return None + if newkey in self._meta.renamed: + newkey = self._meta.renamed[newkey] + return newkey def _fill(self): # BIND SERVER > NORMALIZE > RAW > LAZY @@ -140,21 +124,30 @@ def _fill(self): setattr(self, server_type[0], server_type[1].lower()) except AttributeError: pass - data_ = self._normalize(self._meta.data) + mapping = normalizer_cache.get(self.__class__, dict) + + for attr, renamed in self._meta.renamed.items(): + if renamed != self._meta.server_type: + continue + server = self._meta.data.pop(attr, None) + if server: + setattr(self, self._meta.server_type, server.lower()) + break - if self._meta.server_type in data_: - has_server = True - else: - has_server = False + for key, val in self._meta.data.items(): + try: + attr = mapping[key] + except KeyError: + attr = self._rename(key) + mapping[key] = attr + + if attr is None: + continue - for attr, val in data_.items(): if attr in self._meta.raws: setattr(self, attr, val) elif PyotLazyObject.need_lazy(val): - if has_server: - server = data_[self._meta.server_type] - else: - server = getattr(self, self._meta.server_type) + server = getattr(self, self._meta.server_type) setattr(self, attr, PyotLazyObject(self._meta.types[attr], val, self._meta.server_type, server)) elif attr == self._meta.server_type: setattr(self, attr, val.lower()) @@ -204,6 +197,7 @@ def json(self, pyotify: bool = False, remove_server: bool = True): class PyotContainerObject: class Meta: + # THIS META CLASS IS NOT INHERITED ON CORE, USED ONLY ON CONTAINER server_type: str = "locale" region_list = [] platform_list = [] @@ -263,17 +257,30 @@ class Meta(PyotStaticObject.Meta): platform_list = [] locale_list = [] - async def get(self, sid: str = None, pipeline: str = None): + async def get(self, sid: str = None, pipeline: str = None, ptr_cache: PtrCache = None): '''Awaitable. Get this object from the pipeline.\n `sid` id identifying the session on the pipeline to reuse.\n `pipeline` key identifying the pipeline to execute against.\n + `ptr_cache` intercepts a PtrCache, usage details please refer to documentations.\n ''' self.set_pipeline(pipeline) token = await self.create_token() + + if ptr_cache: + if not isinstance(ptr_cache, PtrCache): + raise TypeError(f"'ptr_cache' receives object of type 'PtrCache', got '{ptr_cache.__class__.__name__}'") + item = ptr_cache.get(token) + if item: + return item + data = await self._meta.pipeline.get(token, sid) data = self._filter(data) self._meta.data = self._transform(data) self._fill() + + if ptr_cache: + ptr_cache.set(token, self) + return self async def post(self, sid: str = None, pipeline: str = None): @@ -307,7 +314,7 @@ def _lazy_set(self, kwargs): self._meta = self.Meta() self._meta.query = {} self._meta.data = {} - self._meta.types = typing_cache.get(self.__class__, partial(get_type_hints, self.__class__)) + self._meta.types = typing_cache.get(self.__class__, self._get_types) self._set_server_type(kwargs) for name, val in kwargs.items(): @@ -334,8 +341,8 @@ def set_pipeline(self, pipeline: str = None): if pipeline is None: return self try: self._meta.pipeline = pipelines[pipeline] - except KeyError: - raise RuntimeError(f"Pipeline '{pipeline}' does not exist, inactive or dead") + except KeyError as e: + raise RuntimeError(f"Pipeline '{pipeline}' does not exist, inactive or dead") from e return self def _parse_camel(self, kwargs) -> Dict: @@ -349,7 +356,7 @@ async def create_token(self, search: str = None) -> PipelineToken: self._get_server() self._refactor() self._validate() - if not hasattr(self._meta, "pipeline"): raise RuntimeError("Pyot pipeline for this model wasn't activated or lost") + if not hasattr(self._meta, "pipeline"): raise RuntimeError("Pyot pipeline for this model is not activated or lost") return PipelineToken(self._meta.pipeline.model, self._meta.server, self._meta.key, self._meta.load, self._meta.query) def _get_rule(self, search): diff --git a/pyot/core/objects_.py b/pyot/core/objects_.py new file mode 100644 index 00000000..71b1b766 --- /dev/null +++ b/pyot/core/objects_.py @@ -0,0 +1,406 @@ +from typing import Dict, List, Mapping, Any, get_type_hints +import pickle +import re +import json + +from pyot.pipeline.core import Pipeline +from pyot.pipeline.token import PipelineToken +from pyot.utils import PtrCache, camelcase, fast_copy +from pyot.pipeline import pipelines + +normalizer_cache = PtrCache() +typing_cache = PtrCache() + + +class PyotLazyObject: + obj: Any + clas: Any + + def __init__(self, clas, obj, server_type, server): + self.clas = clas + self.server_map = [server_type, server] + self.obj = obj + + def __call__(self): + if issubclass(self.clas, PyotCoreObject): + if isinstance(self.obj, list): + li = [] + for obj in self.obj: + instance = self.clas() + instance._meta.server_map = self.server_map + instance._meta.data = instance._transform(obj) + instance._fill() + li.append(instance) + return li + else: + instance = self.clas() + # SERVER MAP WILL GO FIRST THAN OBJECT + instance._meta.server_map = self.server_map + instance._meta.data = instance._transform(self.obj) + instance._fill() + return instance + elif issubclass(self.clas, PyotStaticObject): + if isinstance(self.obj, list): + l = [] + for obj in self.obj: + instance = self.clas(obj) + instance._meta.server_map = self.server_map + l.append(instance._fill()) + return l + else: + instance = self.clas(self.obj) + instance._meta.server_map = self.server_map + return instance._fill() + raise RuntimeError(f"Unable to lazy load '{self.clas}'") + + @staticmethod + def need_lazy(obj): + if isinstance(obj, list) or isinstance(obj, dict): + return True + return False + + +class PyotStaticObject: + + class Meta: + # BE CAREFUL WHEN MANIPULATING MUTABLE OBJECTS + # ALL MUTABLE OBJECTS SHOULD BE OVERRIDDEN ON ITS SUBCLASS ! + server_map: List[str] + types: Dict[str, Any] + data: Dict[str, Any] + raws: List[str] = [] + removed: List[str] = [] + renamed: Dict[str, str] = {} + + region: str = "" + platform: str = "" + locale: str = "" + + def __init__(self, data): + # META CLASS UNIQUE MUTABLE OBJECTS + self._meta = self.Meta() + self._meta.data = data + self._meta.types = typing_cache.get(self.__class__, self._get_types) + + def __getattribute__(self, name): + try: + attr = super().__getattribute__(name) + if isinstance(attr, PyotLazyObject): + obj = attr() + setattr(self, name, obj) + return obj + else: + return attr + except (KeyError, AttributeError): + raise AttributeError(f"'{self.__class__.__name__}' object has no attribute '{name}'") + + def __getitem__(self, item): + return self._meta.data[item] + + def _rename(self, data): + # SNAKECASE > RENAME > REMOVE + new_data = {} + mapping = {} + for attr, val in data.items(): + name = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', attr) + newkey = re.sub('([a-z0-9])([A-Z])', r'\1_\2', name).lower() + if newkey in self._meta.removed: + continue + if newkey in self._meta.renamed: + newkey = self._meta.renamed[newkey] + mapping[attr] = newkey + new_data[newkey] = val + + return new_data, mapping + + def _get_types(self): + types = get_type_hints(self.__class__) + for typ, clas in types.items(): + try: + if clas.__origin__ is list: + types[typ] = clas.__args__[0] + except Exception: + pass + return types + + def _normalize(self, data): + mapping = normalizer_cache.get(self.__class__, dict) + new_data = {} + for attr, val in data.items(): + try: + new_data[mapping[attr]] = val + except KeyError: + new_data, new_mapping = self._rename(data) + mapping.update(new_mapping) + return new_data + for key in self._meta.removed: + new_data.pop(key, None) + return new_data + + def _fill(self): + # BIND SERVER > NORMALIZE > RAW > LAZY + try: + server_type = self._meta.server_map + self._meta.server_type = server_type[0] + setattr(self, server_type[0], server_type[1].lower()) + except AttributeError: pass + + data_ = self._normalize(self._meta.data) + + if self._meta.server_type in data_: + has_server = True + else: + has_server = False + + for attr, val in data_.items(): + if attr in self._meta.raws: + setattr(self, attr, val) + elif PyotLazyObject.need_lazy(val): + if has_server: + server = data_[self._meta.server_type] + else: + server = getattr(self, self._meta.server_type) + setattr(self, attr, PyotLazyObject(self._meta.types[attr], val, self._meta.server_type, server)) + elif attr == self._meta.server_type: + setattr(self, attr, val.lower()) + else: + setattr(self, attr, val) + return self + + def dict(self, pyotify: bool = False, remove_server: bool = True): + ''' + Convert this pyot object to a python dictionary.\n + Set `pyotify` to True to return a dict with the same schema of the pyot object (This is expensive due to recursion).\n + Set `remove_server` to False to not remove the server values (region/platform/locale). + ''' + if not pyotify: + return fast_copy(self._meta.data) # USING PICKLE FOR FASTER COPY + def recursive(obj): + dic = obj.__dict__ + del dic["_meta"] + if remove_server: + dic.pop(self._meta.server_type, None) + for key, val in dic.items(): + if type(val) is PyotLazyObject: + obj = val() + if isinstance(obj, list): + dic[key] = [] + for i in range(len(obj)): + inner = recursive(obj[i]) + dic[key].append(inner) + else: + inner = recursive(obj) + dic[key] = inner + return dic + obj = pickle.loads(pickle.dumps(self)) + return recursive(obj) + + def json(self, pyotify: bool = False, remove_server: bool = True): + ''' + Convert this pyot object to a json string.\n + Set `pyotify` to True to return a json with the same schema of the pyot object (This is expensive due to recursion).\n + Set `remove_server` to False to not remove the server values (region/platform/locale). + ''' + if not pyotify: + return json.dumps(self._meta.data) + return json.dumps(self.dict(pyotify=pyotify, remove_server=remove_server)) + + +class PyotContainerObject: + + class Meta: + # THIS META CLASS IS NOT INHERITED ON CORE, USED ONLY ON CONTAINER + server_type: str = "locale" + region_list = [] + platform_list = [] + locale_list = [] + + region: str = "" + platform: str = "" + locale: str = "" + + def __init__(self, kwargs): + # META CLASS UNIQUE MUTABLE OBJECTS + self._meta = self.Meta() + self._set_server_type(kwargs) + for name, val in kwargs.items(): + if name in ["platform", "region", "locale"] and val is not None: + setattr(self, name, val.lower()) + + def _get_server(self): + for server_type in ["platform", "region", "locale"]: + if self._meta.server_type == server_type: + server = getattr(self, server_type) + list_ = getattr(self._meta, server_type+"_list") + if server.lower() not in list_: + raise ValueError(f"Invalid '{server_type}' value, '{server}' was given \ + {'. Did you activate the settings and set a default value ?' if not server else ''}") + self._meta.server = server.lower() + break + if server_type == "locale": # if server is last and still not found, raise + raise TypeError("Invalid or missing server type was passed as subclass") + return self + + def _set_server_type(self, kwargs): + for server in ["platform", "region", "locale"]: + if server in kwargs: + self._meta.server_type = server + break + if server == "locale": # if server is last and still not found, raise + raise TypeError("Invalid or missing server type was passed as subclass") + + +class PyotCoreObject(PyotStaticObject, PyotContainerObject): + + class Meta(PyotStaticObject.Meta): + # BE CAREFUL WHEN MANIPULATING MUTABLE OBJECTS + # ALL MUTABLE OBJECTS MUST BE OVERRIDDEN ON ITS SUBCLASS ! + pipeline: Pipeline + key: str + server: str + session_id: str = None + load: Mapping[str, Any] + query: Mapping[str, Any] + body: Mapping[str, Any] + server_type: str = "platform" + allow_query: bool = False + rules: Mapping[str, List[str]] = {} + region_list = [] + platform_list = [] + locale_list = [] + + async def get(self, sid: str = None, pipeline: str = None, ptr_cache: PtrCache = None): + '''Awaitable. Get this object from the pipeline.\n + `sid` id identifying the session on the pipeline to reuse.\n + `pipeline` key identifying the pipeline to execute against.\n + `ptr_cache` intercepts a PtrCache, usage details please refer to documentations.\n + ''' + self.set_pipeline(pipeline) + token = await self.create_token() + + if ptr_cache: + if not isinstance(ptr_cache, PtrCache): + raise TypeError(f"'ptr_cache' receives object of type 'PtrCache', got '{ptr_cache.__class__.__name__}'") + item = ptr_cache.get(token) + if item: + return item + + data = await self._meta.pipeline.get(token, sid) + data = self._filter(data) + self._meta.data = self._transform(data) + self._fill() + + if ptr_cache: + ptr_cache.set(token, self) + + return self + + async def post(self, sid: str = None, pipeline: str = None): + '''Awaitable. Post this object to the pipeline.\n + `sid` id identifying the session on the pipeline to reuse.\n + `pipeline` key identifying the pipeline to execute against.\n + ''' + self.set_pipeline(pipeline) + token = await self.create_token() + data = await self._meta.pipeline.post(token, self._meta.body, sid) + data = self._filter(data) + self._meta.data = self._transform(data) + self._fill() + return self + + async def put(self, sid: str = None, pipeline: str = None): + '''Awaitable. Put this object to the pipeline.\n + `sid` id identifying the session on the pipeline to reuse.\n + `pipeline` key identifying the pipeline to execute against.\n + ''' + self.set_pipeline(pipeline) + token = await self.create_token() + data = await self._meta.pipeline.put(token, self._meta.body, sid) + data = self._filter(data) + self._meta.data = self._transform(data) + self._fill() + return self + + def _lazy_set(self, kwargs): + # META CLASS UNIQUE MUTABLE OBJECTS + self._meta = self.Meta() + self._meta.query = {} + self._meta.data = {} + self._meta.types = typing_cache.get(self.__class__, self._get_types) + + self._set_server_type(kwargs) + for name, val in kwargs.items(): + if name in ["platform", "region", "locale"] and val is not None: + self._meta.data[name] = val.lower() + setattr(self, name, val.lower()) + elif val is not None and name != "self": + self._meta.data[name] = val + setattr(self, name, val) + return self + + def query(self, **kwargs): + '''Add query parameters to the object.''' + self._meta.query = self._parse_camel(locals()) + return self + + def body(self, **kwargs): + '''Add body parameters to the object.''' + self._meta.body = self._parse_camel(locals()) + return self + + def set_pipeline(self, pipeline: str = None): + '''Set the pipeline to execute against.''' + if pipeline is None: return self + try: + self._meta.pipeline = pipelines[pipeline] + except KeyError: + raise RuntimeError(f"Pipeline '{pipeline}' does not exist, inactive or dead") + return self + + def _parse_camel(self, kwargs) -> Dict: + '''Parse locals to json compatible camelcased keys''' + return {camelcase(key): val for (key,val) in kwargs.items() if key != "self" and val is not None} + + async def create_token(self, search: str = None) -> PipelineToken: + '''Awaitable. Create a pipeline token that identifies this object (its parameters).''' + await self._clean() + self._get_rule(search) + self._get_server() + self._refactor() + self._validate() + if not hasattr(self._meta, "pipeline"): raise RuntimeError("Pyot pipeline for this model wasn't activated or lost") + return PipelineToken(self._meta.pipeline.model, self._meta.server, self._meta.key, self._meta.load, self._meta.query) + + def _get_rule(self, search): + if len(self._meta.rules) == 0: + raise RuntimeError("This Pyot object is not getable") + for key, attr in self._meta.rules.items(): + if search and search not in key: continue + load = {} + for a in attr: + try: + load[a] = getattr(self, a) + except AttributeError: + break + if len(load) != len(attr): + continue + self._meta.key = key + self._meta.load = load + return self + raise TypeError("Incomplete values to create request token") + + + async def _clean(self): + pass + + def _validate(self): + pass + + def _transform(self, data) -> Dict: + return data + + def _refactor(self): + pass + + def _filter(self, data): + return data diff --git a/pyot/core/objects_pbe.py b/pyot/core/objects_pbe.py deleted file mode 100644 index 736a103f..00000000 --- a/pyot/core/objects_pbe.py +++ /dev/null @@ -1,320 +0,0 @@ -# THIS IS ONLY A CONCEPT, NOT IMPLEMENTED - -from functools import wraps, partial -from typing import Dict, List, Mapping, Any, get_type_hints -import pickle -import re -import json - -from pyot.pipeline.core import Pipeline -from pyot.pipeline.token import PipelineToken -from pyot.utils import PtrCache - -normalizer_cache = PtrCache() -typing_cache = PtrCache() - - -class PyotLazyObject: - obj: Any - clas: Any - - def __init__(self, clas, obj, server_type, server): - try: - if clas.__origin__ is list: - self.clas = clas.__args__[0] - except Exception: - self.clas = clas - self.server_map = [server_type, server] - self.obj = obj - - def __call__(self): - if issubclass(self.clas, PyotCoreObject): - if isinstance(self.obj, list): - l = [] - for obj in self.obj: - data = obj.pop("data") - instance = self.clas(**obj) - instance.meta.data = data - instance.meta.server_map = self.server_map - instance._fill() - l.append(instance) - return l - else: - # pop the data - data = self.obj.pop("data") - # obj are the arguments - instance = self.clas(**self.obj) - # insert the data and fill - instance.meta.data = data - instance.meta.server_map = self.server_map - instance._fill() - return instance - elif issubclass(self.clas, PyotStaticObject): - if isinstance(self.obj, list): - l = [] - for obj in self.obj: - instance = self.clas(obj) - instance.meta.server_map = self.server_map - l.append(instance._fill()) - return l - else: - instance = self.clas(self.obj) - instance.meta.server_map = self.server_map - return instance._fill() - raise RuntimeError(f"Unable to lazy load '{self.clas}'") - - -class PyotStaticObject: - - class Meta: - # BE CAREFUL WHEN MANIPULATING MUTABLE OBJECTS - # ALL MUTABLE OBJECTS SHOULD BE OVERRIDDEN ON ITS SUBCLASS ! - server_map: List[str] - types: Dict[str, Any] - data: Dict[str, Any] - ata: Dict[str, Any] - raws: List[str] = [] - removed: List[str] = [] - renamed: Dict[str, str] = {} - - region: str = "" - platform: str = "" - locale: str = "" - - def __init__(self, data): - # META CLASS UNIQUE MUTABLE OBJECTS - self.meta = self.Meta() - self.meta.data = data - self.meta.ata = {} - self.meta.types = typing_cache.get(self.__class__, partial(get_type_hints, self.__class__)) - - def __getattribute__(self, name): - if name == "meta": - return super().__getattribute__(name) - try: - val = self.meta.ata[name] - if isinstance(val, list) or isinstance(val, dict): - if name in self.meta.raws: - return val - server = self.meta.ata[self.meta.server_type] - self.meta.ata[name] = PyotLazyObject(self.meta.types[name], val, self.meta.server_type, server)() - return self.meta.ata[name] - return val - except (KeyError, AttributeError): - return super().__getattribute__(name) - - def _rename(self, data): - # SNAKECASE > RENAME > REMOVE - new_data = {} - mapping = {} - for attr, val in data.items(): - name = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', attr) - newkey = re.sub('([a-z0-9])([A-Z])', r'\1_\2', name).lower() - if newkey in self.meta.removed: - continue - if newkey in self.meta.renamed: - newkey = self.meta.renamed[newkey] - mapping[attr] = newkey - new_data[newkey] = val - - return new_data, mapping - - def _normalize(self, data): - mapping = normalizer_cache.get(self.__class__, dict) - new_data = {} - for attr, val in data.items(): - try: - new_data[mapping[attr]] = val - except KeyError: - new_data, new_mapping = self._rename(data) - mapping.update(new_mapping) - return new_data - for key in self.meta.removed: - new_data.pop(key, None) - return new_data - - def _fill(self): - # BIND SERVER > NORMALIZE > RAW > LAZY - try: - server_type = self.meta.server_map - self.meta.server_type = server_type[0] - setattr(self, server_type[0], server_type[1].lower()) - except AttributeError: pass - - data_ = self._normalize(self.meta.data) - - if self.meta.server_type in data_: - data_[self.meta.server_type] = data_[self.meta.server_type].lower() - setattr(self, self.meta.server_type, data_[self.meta.server_type].lower()) - - self.meta.ata = data_ - self.meta.ata.update(self.__dict__) - # for attr, val in data_.items(): - # if attr in self.meta.raws: - # setattr(self, attr, val) - # elif PyotLazyObject.need_lazy(val): - # if has_server: - # server = data_[self.meta.server_type] - # else: - # server = getattr(self, self.meta.server_type) - # setattr(self, attr, PyotLazyObject(self.meta.types[attr], val, self.meta.server_type, server)) - # elif attr == self.meta.server_type: - # setattr(self, attr, val.lower()) - # else: - # setattr(self, attr, val) - return self - - def dict(self, pyotify: bool = False, remove_server: bool = True): - ''' - Convert this pyot object to a python dictionary.\n - Set `pyotify` to True to return a dict with the same schema of the pyot object (This is expensive due to recursion).\n - Set `remove_server` to False to not remove the server values (region/platform/locale). - ''' - if not pyotify: - return pickle.loads(pickle.dumps(self.meta.data)) # USING PICKLE FOR FASTER COPY - def recursive(obj): - dic = obj.__dict__ - del dic["meta"] - if remove_server: - dic.pop(self.meta.server_type, None) - for key, val in dic.items(): - if type(val) is PyotLazyObject: - obj = val() - if isinstance(obj, list): - dic[key] = [] - for i in range(len(obj)): - inner = recursive(obj[i]) - dic[key].append(inner) - else: - inner = recursive(obj) - dic[key] = inner - return dic - obj = pickle.loads(pickle.dumps(self)) - return recursive(obj) - - def json(self, pyotify: bool = False, remove_server: bool = True): - ''' - Convert this pyot object to a json string.\n - Set `pyotify` to True to return a json with the same schema of the pyot object (This is expensive due to recursion).\n - Set `remove_server` to False to not remove the server values (region/platform/locale). - ''' - if not pyotify: - return json.dumps(self.meta.data) - return json.dumps(self.dict(pyotify=pyotify, remove_server=remove_server)) - - -class PyotCoreObject(PyotStaticObject): - - class Meta(PyotStaticObject.Meta): - # BE CAREFUL WHEN MANIPULATING MUTABLE OBJECTS - # ALL MUTABLE OBJECTS SHOULD BE OVERRIDDEN ON ITS SUBCLASS ! - pipeline: Pipeline - key: str - server: str - session_id: str = None - load: Mapping[str, Any] - query: Mapping[str, Any] - server_type: str = "platform" - allow_query: bool = False - rules: Mapping[str, List[str]] = {} - region_list = [] - platform_list = [] - locale_list = [] - - async def get(self, sid: str = None): - '''Awaitable. Get this object from the pipeline.\n - `sid` may be passed to reuse a session on the pipeline.''' - token = await self.create_token() - data = await self.meta.pipeline.get(token, sid) - data = self.filter(data) - self.meta.data = self._transform(data) - self._fill() - return self - - def _lazy_set(self, kwargs): - # META CLASS UNIQUE MUTABLE OBJECTS - self.meta = self.Meta() - self.meta.query = {} - self.meta.data = {} - self.meta.types = typing_cache.get(self.__class__, partial(get_type_hints, self.__class__)) - - for server in ["platform", "region", "locale"]: - if server in kwargs: - self.meta.server_type = server - break - if server == "locale": # if server is last and still not found, raise - raise RuntimeError("Invalid or missing server type was passed as subclass") - for name, val in kwargs.items(): - if name in ["platform", "region", "locale"] and val is not None: - self.meta.data[name] = val.lower() - setattr(self, name, val.lower()) - elif val is not None and name != "self": - self.meta.data[name] = val - setattr(self, name, val) - return self - - def query(self, **kwargs): - '''Add query parameters to the object.''' - if not self.meta.allow_query: - raise RuntimeError("This Pyot object does not accept queries") - self.meta.query = self._parse_query(locals()) - return self - - def to_camel_case(self, snake_str): - components = snake_str.split('_') - return components[0] + ''.join(x.title() for x in components[1:]) - - def _parse_query(self, kwargs) -> Dict: - return {self.to_camel_case(key): val for (key,val) in kwargs.items() if key != "self" and val is not None} - - async def create_token(self, search: str = None) -> PipelineToken: - '''Awaitable. Create a pipeline token that identifies this object (its parameters).''' - await self._clean() - self._get_rule(search) - self._get_server() - self._refactor() - if not hasattr(self.meta, "pipeline"): raise RuntimeError("Pyot pipeline for this model wasn't activated or lost") - return PipelineToken(self.meta.pipeline.model, self.meta.server, self.meta.key, self.meta.load, self.meta.query) - - def _get_rule(self, search): - if len(self.meta.rules) == 0: - raise RuntimeError("This Pyot object is not getable") - for key, attr in self.meta.rules.items(): - if search and search not in key: continue - load = {} - for a in attr: - try: - load[a] = getattr(self, a) - except AttributeError: - break - if len(load) != len(attr): - continue - self.meta.key = key - self.meta.load = load - return self - raise ValueError("Incomplete values to create request token") - - def _get_server(self): - for server_type in ["platform", "region", "locale"]: - if self.meta.server_type == server_type: - server = getattr(self, server_type) - list_ = getattr(self.meta, server_type+"_list") - if server.lower() not in list_: - raise ValueError(f"Invalid '{server_type}' value, '{server}' was given") - self.meta.server = server.lower() - break - if server_type == "locale": # if server is last and still not found, raise - raise RuntimeError("Invalid or missing server type was passed as subclass") - return self - - async def _clean(self): - pass - - def _transform(self, data) -> Dict: - return data - - def _refactor(self): - pass - - def filter(self, data): - return data diff --git a/pyot/limiters/core.py b/pyot/limiters/core.py index e6b318f3..fc7d7116 100644 --- a/pyot/limiters/core.py +++ b/pyot/limiters/core.py @@ -191,8 +191,8 @@ async def put_stream(self, fetched: dict, server: str, method: str, token: Limit if not r_method_rate[i][0] or method_count[i][0] < r_method_rate[i][0]: r_method_rate[i][0] = method_count[i][0] for i in range(2): - app_top = date + timedelta(seconds=r_app_rate[i][3]) - method_top = date + timedelta(seconds=r_method_rate[i][3]) + app_top = date + timedelta(seconds=r_app_rate[i][3] + 1) + method_top = date + timedelta(seconds=r_method_rate[i][3] + 1) if app_top < r_app_time[i] or (token.flag_app and r_app_time[i] < now): r_app_time[i] = app_top if method_top < r_method_time[i] or (token.flag_method and r_method_time[i] < now): diff --git a/pyot/management/__init__.py b/pyot/management/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/pyot/management/__main__.py b/pyot/management/__main__.py new file mode 100644 index 00000000..3c63c8e5 --- /dev/null +++ b/pyot/management/__main__.py @@ -0,0 +1,5 @@ +from .scripts import main + + +if __name__ == '__main__': + main() diff --git a/pyot/management/scripts.py b/pyot/management/scripts.py new file mode 100644 index 00000000..7f779395 --- /dev/null +++ b/pyot/management/scripts.py @@ -0,0 +1,24 @@ +import os +import sys +from distutils.dir_util import copy_tree + +def startproject(dirname): + src = os.path.dirname(os.path.realpath(__file__)) + if '\\' in src: + src += '\\startproject_files' + else: + src += '/startproject_files' + copy_tree(src, dirname) + + +scripts = { + 'startproject': startproject, +} + +def main(): + vals = sys.argv[1:] + if len(vals) < 2: + raise ValueError('At least 2 argument needs to be provided') + command = vals[0] + args = vals[1:] + scripts[command](*args) diff --git a/pyot/management/startproject_files/__init__.py b/pyot/management/startproject_files/__init__.py new file mode 100644 index 00000000..74c54180 --- /dev/null +++ b/pyot/management/startproject_files/__init__.py @@ -0,0 +1,8 @@ + +# Imports the settings to use. +try: + from . import settings +except KeyError as e: + raise KeyError(str(e)[1:-1] + ", Please check if issue is related to environment variables and reopen the console") from e + +# Other imports if needed... diff --git a/pyot/management/startproject_files/__main__.py b/pyot/management/startproject_files/__main__.py new file mode 100644 index 00000000..27d3a9bb --- /dev/null +++ b/pyot/management/startproject_files/__main__.py @@ -0,0 +1,9 @@ +# Entry point of this module +# Defaults to executing the task manager +# To run this module: `python -m `. + +import sys +from .manage import execute_task_from_args + +if __name__ == "__main__": + execute_task_from_args() diff --git a/pyot/management/startproject_files/manage.py b/pyot/management/startproject_files/manage.py new file mode 100644 index 00000000..b67b6034 --- /dev/null +++ b/pyot/management/startproject_files/manage.py @@ -0,0 +1,21 @@ +# Default task manager for pyot projects. +# Feel free to change this file to your wanted behavior. + +import sys +import asyncio + +# Loads declared callables +from . import tasks + + +def execute_task_from_args(): + try: + funcname = sys.argv[1] + args = sys.argv[2:] if len(sys.argv) > 2 else [] + except IndexError as e: + raise Exception('Missing callable name for execution') from e + func = getattr(tasks, funcname) + if asyncio.iscoroutinefunction(func): + asyncio.run(func(*args)) + else: + func(*args) diff --git a/pyot/management/startproject_files/settings.py b/pyot/management/startproject_files/settings.py new file mode 100644 index 00000000..dbf87049 --- /dev/null +++ b/pyot/management/startproject_files/settings.py @@ -0,0 +1,40 @@ +# Default settings for pyot projects. +# Change these settings according to your needs. + +import platform +import os + + +# Fix: Windows `asyncio.run()` will throw `RuntimeError: Event loop is closed`. +# Refer: https://github.com/aio-libs/aiohttp/issues/4324 + +if platform.system() == 'Windows': + from asyncio.proactor_events import _ProactorBasePipeTransport + from pyot.utils.internal import silence_event_loop_closed + _ProactorBasePipeTransport.__del__ = silence_event_loop_closed(_ProactorBasePipeTransport.__del__) + + +# Pyot documentations for settings +# https://paaksing.github.io/Pyot/core/settings.html + +# Pyot documentations for pipeline stores +# https://paaksing.github.io/Pyot/stores/ + +from pyot.core import Settings + + +Settings( + MODEL = "LOL", + DEFAULT_PLATFORM = "NA1", + DEFAULT_REGION = "AMERICAS", + DEFAULT_LOCALE= "EN_US", + PIPELINE = [ + {"BACKEND": "pyot.stores.Omnistone"}, + {"BACKEND": "pyot.stores.MerakiCDN"}, + {"BACKEND": "pyot.stores.CDragon"}, + { + "BACKEND": "pyot.stores.RiotAPI", + "API_KEY": os.environ["RIOT_API_KEY"], # API KEY + } + ] +).activate() # <- DON'T FORGET TO ACTIVATE THE SETTINGS diff --git a/pyot/management/startproject_files/tasks.py b/pyot/management/startproject_files/tasks.py new file mode 100644 index 00000000..e6007ba9 --- /dev/null +++ b/pyot/management/startproject_files/tasks.py @@ -0,0 +1,19 @@ +# Default startup tasks declaration file for pyot projects. +# Callables declared in this file will be accessible by the task manager. + +from pyot.models import lol +from pyot.core.exceptions import NotFound + + +async def summoner_level(name, platform): + '''Get summoner level by name and platform''' + + # Pyot Model: lol + # Pyot Core Object: Summoner + # Refer: https://paaksing.github.io/Pyot/models/lol_summoner.html + + try: + summoner = await lol.Summoner(name=name, platform=platform).get() + print(summoner.name, 'in', summoner.platform.upper(), 'is level', summoner.level) + except NotFound: + print('Summoner not found') diff --git a/pyot/models/lol/merakiitem.py b/pyot/models/lol/merakiitem.py index caddb75f..07ceabf1 100644 --- a/pyot/models/lol/merakiitem.py +++ b/pyot/models/lol/merakiitem.py @@ -32,6 +32,8 @@ class MerakiItemStatData(PyotStatic): mana: MerakiItemStatDetailData mana_regen: MerakiItemStatDetailData movespeed: MerakiItemStatDetailData + ability_haste: MerakiItemStatDetailData + omnivamp: MerakiItemStatDetailData class MerakiItemPassiveData(PyotStatic): @@ -71,6 +73,7 @@ class MerakiItem(PyotCore): name: str id: int tier: int + rank: List[str] builds_from_ids: List[int] builds_into_ids: List[int] no_effects: bool @@ -87,7 +90,7 @@ class MerakiItem(PyotCore): class Meta(PyotCore.Meta): rules = {"meraki_item_by_id": ["id"]} - raws = ["builds_from_ids", "builds_into_ids", "nicknames"] + raws = ["builds_from_ids", "builds_into_ids", "nicknames", "rank"] renamed = {"builds_from": "builds_from_ids", "builds_into": "builds_into_ids", "required_champion": "required_champion_key"} def __init__(self, id: int = None): @@ -137,4 +140,4 @@ def required_champion(self) -> "Champion": @property def meraki_required_champion(self) -> "MerakiChampion": from .merakichampion import MerakiChampion - return MerakiChampion(id=self.required_champion_key) \ No newline at end of file + return MerakiChampion(id=self.required_champion_key) diff --git a/pyot/models/lol/tournament.py b/pyot/models/lol/tournament.py index abd3845f..63d3bf7a 100644 --- a/pyot/models/lol/tournament.py +++ b/pyot/models/lol/tournament.py @@ -23,7 +23,7 @@ def __init__(self, region: str = None): def body(self, region: str, url: str): '''Add body parameters to the object.''' - if not url.startswith("https://") or not url.startswith("http://"): + if not url.startswith("https://") and not url.startswith("http://"): raise TypeError("url should be well-formed, starting with an http protocol") if region not in ["BR", "EUNE", "EUW", "JP", "LAN", "LAS", "NA", "OCE", "PBE", "RU", "TR"]: raise TypeError(f"Invalid region '{region}' parameter, region in tournament-v4 uses client keys (NA, EUW, BR, LAN, ...)") diff --git a/pyot/pipeline/core.py b/pyot/pipeline/core.py index 5b3cdced..6a2dc1e8 100644 --- a/pyot/pipeline/core.py +++ b/pyot/pipeline/core.py @@ -25,7 +25,7 @@ async def get(self, token: PipelineToken, sid: str = None): response = await store.get(token, session=session) found_in = store except (NotImplementedError, NotFound, NotFindable) as e: - if self.stores[-1] == store or store.store_type == "SERVICE" and not isinstance(e, NotFindable): + if self.stores[-1] == store or store.store_type == "SERVICE" and isinstance(e, NotFound): raise continue break @@ -48,7 +48,7 @@ async def post(self, token: PipelineToken, body: Any, sid: str = None): try: response = await store.post(token, body, session=session) except (NotImplementedError, NotFound, NotFindable) as e: - if self.stores[-1] == store or store.store_type == "SERVICE" and not isinstance(e, NotFindable): + if self.stores[-1] == store or store.store_type == "SERVICE" and isinstance(e, NotFound): raise continue break @@ -61,7 +61,7 @@ async def put(self, token: PipelineToken, body: Any, sid: str = None): try: response = await store.put(token, body, session=session) except (NotImplementedError, NotFound, NotFindable) as e: - if self.stores[-1] == store or store.store_type == "SERVICE" and not isinstance(e, NotFindable): + if self.stores[-1] == store or store.store_type == "SERVICE" and isinstance(e, NotFound): raise continue break diff --git a/pyot/stores/omnistone.py b/pyot/stores/omnistone.py index 5b323212..b7aa2206 100644 --- a/pyot/stores/omnistone.py +++ b/pyot/stores/omnistone.py @@ -35,7 +35,7 @@ async def set(self, token: PipelineToken, value: Any) -> None: if timeout != -1: timeout = datetime.timedelta(seconds=timeout) if await self._allowed(): - value = pickle.loads(pickle.dumps(value)) + value = pickle.dumps(value) self._data[token] = (value, timeout, datetime.datetime.now(), datetime.datetime.now()) LOGGER.log(self._log_level, f"[Trace: {self._game.upper()} > Omnistone] SET: {self._log_template(token)}") if len(self._data) > self._max_entries and await self._allowed(): @@ -60,7 +60,7 @@ async def get(self, token: PipelineToken, expiring: bool = False, session = None now = datetime.datetime.now() if timeout == -1: self._data[token] = (item, timeout, entered, now) - item = pickle.loads(pickle.dumps(item)) + item = pickle.loads(item) return item elif now > entered + timeout: @@ -71,7 +71,7 @@ async def get(self, token: PipelineToken, expiring: bool = False, session = None else: self._data[token] = (item, timeout, entered, now) - item = pickle.loads(pickle.dumps(item)) + item = pickle.loads(item) return item async def delete(self, token: PipelineToken) -> None: diff --git a/pyot/utils/__init__.py b/pyot/utils/__init__.py index 895b912b..4a87c04a 100644 --- a/pyot/utils/__init__.py +++ b/pyot/utils/__init__.py @@ -3,7 +3,9 @@ from .cdragon import cdragon_url, cdragon_sanitize, tft_url, tft_item_sanitize, tft_champ_sanitize from .objects import ptr_cache, frozen_generator, PtrCache, FrozenGenerator from .dicts import multi_defaultdict, redis_defaultdict, MultiDefaultDict, RedisDefaultDict -from .common import snakecase, camelcase, shuffle_list, loop_run, thread_run, import_class, fast_copy, pytify, bytify +from .common import snakecase, camelcase, shuffle_list, loop_run, thread_run, import_class, fast_copy, pytify, bytify, inherit_docstrings +from .internal import silence_event_loop_closed from .locks import SealLock, RedisLock from .time import timeit, atimeit from .lor import batch_to_ccac + diff --git a/pyot/utils/champion.py b/pyot/utils/champion.py index 357379bb..169decdd 100644 --- a/pyot/utils/champion.py +++ b/pyot/utils/champion.py @@ -31,44 +31,53 @@ async def _gather_summary(cache): cache.set(key, val) +# IMPORTANT: _gather_summary() gathers all values, that's why it isn't passed as default. + + async def champion_id_by_key(value): '''Convert champion key to id''' - try: data = _utils_inner_cache.get("champion_id_by_key") - except KeyError: await _gather_summary(_utils_inner_cache) data = _utils_inner_cache.get("champion_id_by_key") + if data is None: + await _gather_summary(_utils_inner_cache) + data = _utils_inner_cache.get("champion_id_by_key") return data[value] async def champion_id_by_name(value): '''Convert champion name to id''' - try: data = _utils_inner_cache.get("champion_id_by_name") - except KeyError: await _gather_summary(_utils_inner_cache) data = _utils_inner_cache.get("champion_id_by_name") + if data is None: + await _gather_summary(_utils_inner_cache) + data = _utils_inner_cache.get("champion_id_by_name") return data[value] async def champion_key_by_id(value): '''Convert champion id to key''' - try: data = _utils_inner_cache.get("champion_key_by_id") - except KeyError: await _gather_summary(_utils_inner_cache) data = _utils_inner_cache.get("champion_key_by_id") + if data is None: + await _gather_summary(_utils_inner_cache) + data = _utils_inner_cache.get("champion_key_by_id") return data[value] async def champion_key_by_name(value): '''Convert champion name to key''' - try: data = _utils_inner_cache.get("champion_key_by_name") - except KeyError: await _gather_summary(_utils_inner_cache) data = _utils_inner_cache.get("champion_key_by_name") + if data is None: + await _gather_summary(_utils_inner_cache) + data = _utils_inner_cache.get("champion_key_by_name") return data[value] async def champion_name_by_id(value): '''Convert champion id to name''' - try: data = _utils_inner_cache.get("champion_name_by_id") - except KeyError: await _gather_summary(_utils_inner_cache) data = _utils_inner_cache.get("champion_name_by_id") + if data is None: + await _gather_summary(_utils_inner_cache) + data = _utils_inner_cache.get("champion_name_by_id") return data[value] async def champion_name_by_key(value): '''Convert champion key to name''' - try: data = _utils_inner_cache.get("champion_name_by_key") - except KeyError: await _gather_summary(_utils_inner_cache) data = _utils_inner_cache.get("champion_name_by_key") + if data is None: + await _gather_summary(_utils_inner_cache) + data = _utils_inner_cache.get("champion_name_by_key") return data[value] diff --git a/pyot/utils/common.py b/pyot/utils/common.py index 04981ec7..8183c623 100644 --- a/pyot/utils/common.py +++ b/pyot/utils/common.py @@ -1,3 +1,4 @@ +from inspect import getmembers, isfunction from typing import List, Any from importlib import import_module import asyncio @@ -16,6 +17,15 @@ async def thread_run(func): return await loop.run_in_executor(None, func) +def inherit_docstrings(cls): + for name, func in getmembers(cls, isfunction): + if func.__doc__: continue + for parent in cls.__mro__[1:]: + if hasattr(parent, name): + func.__doc__ = getattr(parent, name).__doc__ + return cls + + def snakecase(attr: str) -> str: '''Convert string to python snakecase.''' name = re.sub('(.)([A-Z][a-z]+)', r'\1_\2', attr) diff --git a/pyot/utils/internal.py b/pyot/utils/internal.py new file mode 100644 index 00000000..e664c20a --- /dev/null +++ b/pyot/utils/internal.py @@ -0,0 +1,12 @@ +from functools import wraps + +def silence_event_loop_closed(func): + '''Silences the Exception `RuntimeError: Event loop is closed` in a class method.''' + @wraps(func) + def wrapper(self, *args, **kwargs): + try: + return func(self, *args, **kwargs) + except RuntimeError as e: + if str(e) != 'Event loop is closed': + raise + return wrapper diff --git a/pyot/utils/objects.py b/pyot/utils/objects.py index aea9ac07..f1ca2429 100644 --- a/pyot/utils/objects.py +++ b/pyot/utils/objects.py @@ -31,7 +31,7 @@ def get(self, name: str, func = None): return data[0] except KeyError: if func is None: - raise + return None response = func() self.set(name, response) return response @@ -54,7 +54,7 @@ async def aget(self, name: str, coro = None): return data[0] except KeyError: if coro is None: - raise + return None response = await coro self.set(name, response) return response diff --git a/setup.py b/setup.py index 59d58809..74458fe9 100644 --- a/setup.py +++ b/setup.py @@ -28,14 +28,14 @@ setup( name="pyot", - version="1.2.0", + version="2.0.0", author="Paaksing", author_email="paaksingtech@gmail.com", url="https://github.com/paaksing/Pyot", - description="2020 High level Python framework for the Riot Games API, support for AsyncIO and Django", + description="AsyncIO based high level Python framework for the Riot Games API that encourages rapid development and clean, pragmatic design.", long_description=long_description, long_description_content_type='text/markdown', - keywords=["Riot Games", "League of Legends", "Teamfight Tactics", "Valorant", "Legends of Runeterra", "API", "REST", "Django"], + keywords=["Riot Games", "League of Legends", "Teamfight Tactics", "Valorant", "Legends of Runeterra", "API", "REST", "Django", "asyncio"], classifiers=[ "Development Status :: 5 - Production/Stable", "Programming Language :: Python :: 3.7", @@ -54,5 +54,10 @@ packages=find_packages(exclude=("test","test_djot")), zip_safe=True, install_requires=install_requires, - include_package_data=True + include_package_data=True, + entry_points={ + 'console_scripts': [ + 'pyot=pyot.management.scripts:main' + ] + } ) diff --git a/test/__init__.py b/test/__init__.py index 6dd35dbd..0c22bf41 100644 --- a/test/__init__.py +++ b/test/__init__.py @@ -10,9 +10,8 @@ DEFAULT_LOCALE= "EN_US", PIPELINE = [ { - "BACKEND": "pyot.stores.MongoDB", + "BACKEND": "pyot.stores.Omnistone", "LOG_LEVEL": 30, - "DB": 'pyot_lol', "EXPIRATIONS": { "summoner_v4_by_name": 100, } @@ -21,6 +20,10 @@ "BACKEND": "pyot.stores.RedisCache", "LOG_LEVEL": 30, "DB": 1, + "EXPIRATIONS": { + "match_v4_match": 600, + "match_v4_timeline": 600, + } }, { "BACKEND": "pyot.stores.MerakiCDN", diff --git a/test/__main__.py b/test/__main__.py index cdcffa92..dab69242 100644 --- a/test/__main__.py +++ b/test/__main__.py @@ -1,12 +1,9 @@ from pyot.utils import loop_run, timeit from .manual_test_1 import pull_dev_key_limit, sync_dev_key_limit -from .manual_test_2 import pull_matchlist from .speed_test_1 import iterate_match_events import cProfile -# cProfile.run("loop_run(pull_matchlist())") -loop_run(pull_matchlist()) # loop_run(pull_dev_key_limit()) -# print(timeit(iterate_match_events)) -# cProfile.run("timeit(iterate_match_events, 100)") +print(timeit(iterate_match_events)) +cProfile.run("timeit(iterate_match_events, 100)") diff --git a/test/manual_test_2.py b/test/manual_test_2.py deleted file mode 100644 index 42dc750a..00000000 --- a/test/manual_test_2.py +++ /dev/null @@ -1,42 +0,0 @@ -from typing import List -from datetime import datetime, timedelta -from pyot.models import lol -from pyot.core import Queue, Gatherer, exceptions -from pyot.utils import FrozenGenerator, shuffle_list - - -async def match_history(queue: Queue, matchlist: set, regions: set, summoner: lol.Summoner, begin: timedelta): - summoner = await summoner.get(sid=queue.sid) - try: - history = await summoner.match_history.query(begin_time=begin).get(sid=queue.sid) - for match in history.matches: - matchlist.add(match.match_timeline) - regions.add(match.platform) - except exceptions.NotFound: - pass - - -async def pull_matchlist(): - platforms = ["la1", "na1"] - matchlist = set() - regions = set() - started = datetime.now() - async with Queue(log_level=30) as queue: # type: Queue - for p in platforms: - await queue.put(lol.MasterLeague(queue="RANKED_SOLO_5x5", platform=p).get(sid=queue.sid)) - await queue.put(lol.DivisionLeague(queue="RANKED_SOLO_5x5", division="I", tier="DIAMOND", platform=p).get(sid=queue.sid)) - - leagues = await queue.join() # type: List[lol.ChallengerLeague] - _summoners = [] - for league in leagues: - for entry in league.entries: - _summoners.append(entry.summoner) - summoners = FrozenGenerator(shuffle_list(_summoners, "platform")) - - begin = round((datetime.now() - timedelta(days=3)).timestamp()*1000) - for summoner in summoners: - await queue.put(match_history(queue, matchlist, regions, summoner, begin)) - await queue.join() - print(len(_summoners)) - print(regions) - print(datetime.now() - started) \ No newline at end of file diff --git a/test/speed_test_1.py b/test/speed_test_1.py index af1395c7..2413c80f 100644 --- a/test/speed_test_1.py +++ b/test/speed_test_1.py @@ -4,5 +4,6 @@ async def iterate_match_events(): match = await lol.MatchTimeline(id=3442099474).get() for team in match.teams: for p in team.participants: - for event in p.timeline.dict()["events"]: + # for event in p.timeline["events"]: + for event in p.timeline.events: pass diff --git a/test/test_tft_static.py b/test/test_tft_static.py index 9a3af5ee..547b4c27 100644 --- a/test/test_tft_static.py +++ b/test/test_tft_static.py @@ -19,7 +19,7 @@ async def async_champions(): v.dict(pyotify=True) async def async_champion(): - val = await tft.Champion(key="TFT3_Darius").get() + val = await tft.Champion(key="TFT4_Aatrox").get() val.dict(pyotify=True) @@ -34,12 +34,12 @@ async def async_item(): async def async_traits(): - val = await tft.Traits(set=3, locale="zh_cn").get() + val = await tft.Traits(set=4, locale="zh_cn").get() for v in val: v.dict(pyotify=True) async def async_trait(): - val = await tft.Trait(key="Battlecast", set=3).get() + val = await tft.Trait(key="Divine", set=4).get() val.dict(pyotify=True)