Skip to content
Permalink
Browse files

PostgreSQL driver, tests against DB backends, and general drivers cle…

…anup (#2723)

* PostgreSQL driver and general drivers cleanup

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Make tests pass

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Add black --target-version flag in make.bat

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Rewrite postgres driver

Most of the logic is now in PL/pgSQL.

This completely avoids the use of Python f-strings to format identifiers into queries. Although an SQL-injection attack would have been impossible anyway (only the owner would have ever had the ability to do that), using PostgreSQL's format() is more reliable for unusual identifiers. Performance-wise, I'm not sure whether this is an improvement, but I highly doubt that it's worse.

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Reformat

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Fix PostgresDriver.delete_all_data()

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Clean up PL/pgSQL code

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* More PL/pgSQL cleanup

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* PL/pgSQL function optimisations

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Ensure compatibility with PostgreSQL 10 and below

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* More/better docstrings for PG functions

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Fix typo in docstring

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Return correct value on toggle()

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Use composite type for PG function parameters

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Fix JSON driver's Config.clear_all()

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Correct description for Mongo tox recipe

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Fix linting errors

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Update dep specification after merging bumpdeps

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Add towncrier entries

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Update from merge

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Mention [postgres] extra in install docs

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Support more connection options and use better defaults

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Actually pass PG env vars in tox

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>

* Replace event trigger with manual DELETE queries

Signed-off-by: Toby Harradine <tobyharradine@gmail.com>
  • Loading branch information...
Tobotimus authored and mikeshardmind committed Aug 27, 2019
1 parent 57fa29d commit d1a46acc9a55f95f31303a32e1059354aec3258d
@@ -5,14 +5,10 @@ notifications:
email: false

python:
- 3.7.2
- 3.7.3
env:
global:
- PIPENV_IGNORE_VIRTUALENVS=1
matrix:
- TOXENV=py
- TOXENV=docs
- TOXENV=style

install:
- pip install --upgrade pip tox
@@ -22,6 +18,19 @@ script:

jobs:
include:
- env: TOXENV=py
- env: TOXENV=docs
- env: TOXENV=style
- env: TOXENV=postgres
services: postgresql
addons:
postgresql: "10"
before_script:
- psql -c 'create database red_db;' -U postgres
- env: TOXENV=mongo
services: mongodb
before_script:
- mongo red_db --eval 'db.createUser({user:"red",pwd:"red",roles:["readWrite"]});'
# These jobs only occur on tag creation if the prior ones succeed
- stage: PyPi Deployment
if: tag IS present
@@ -1,8 +1,8 @@
# Python Code Style
reformat:
black -l 99 `git ls-files "*.py"`
black -l 99 --target-version py37 `git ls-files "*.py"`
stylecheck:
black --check -l 99 `git ls-files "*.py"`
black --check -l 99 --target-version py37 `git ls-files "*.py"`

# Translations
gettext:
@@ -0,0 +1 @@
Added a config driver for PostgreSQL
@@ -0,0 +1,32 @@
Changes to the ``redbot.core.drivers`` package:

- The modules inside the ``redbot.core.drivers`` package no longer have the ``red_`` prefix in front of their names.
- All driver classes are now directly accessible as attributes to the ``redbot.core.drivers`` package.
- :func:`get_driver`'s signature has been changed.
- :func:`get_driver` can now use data manager to infer the backend type if it is not supplied as an argument.
- :func:`get_driver_class` has been added.

Changes to the :class:`BaseDriver` and :class:`JsonDriver` classes class:

- :meth:`BaseDriver.get_config_details` is an now abstract staticmethod.
- :meth:`BaseDriver.initialize` and :meth:`BaseDriver.teardown` are two new abstract coroutine classmethods.
- :meth:`BaseDriver.delete_all_data` is a new concrete (but overrideable) coroutine instance method.
- :meth:`BaseDriver.aiter_cogs` is a new abstract asynchronous iterator method.
- :meth:`BaseDriver.migrate_to` is a new concrete coroutine classmethod.
- :class:`JsonDriver` no longer requires the data path when constructed and will infer the data path from data manager.

Changes to the :class:`IdentifierData` class and :class:`ConfigCategory` enum:

- ``IdentifierData.custom_group_data`` has been replaced by :attr:`IdentifierData.primary_key_len`.
- :meth:`ConfigCategory.get_pkey_info` is a new classmethod.

Changes to the migration and backup system:

- All code in the ``redbot.setup`` script, excluding that regarding MongoV1, is now virtually backend-agnostic.
- All code in the ``[p]backup`` is now backend-agnostic.
- :func:`redbot.core.config.migrate` is a new coroutine function.
- All a new driver needs to do now to be compatible with migrations and backups is to implement the :class:`BaseDriver` ABC.

Enhancements to unit tests:

- New tox recipes have been added for testing against Mongo and Postgres backends. See the ``tox.ini`` file for clues on how to run them.
@@ -421,15 +421,15 @@ Driver Reference

Base Driver
^^^^^^^^^^^
.. autoclass:: redbot.core.drivers.red_base.BaseDriver
.. autoclass:: redbot.core.drivers.BaseDriver
:members:

JSON Driver
^^^^^^^^^^^
.. autoclass:: redbot.core.drivers.red_json.JSON
.. autoclass:: redbot.core.drivers.JsonDriver
:members:

Mongo Driver
^^^^^^^^^^^^
.. autoclass:: redbot.core.drivers.red_mongo.Mongo
.. autoclass:: redbot.core.drivers.MongoDriver
:members:
@@ -200,6 +200,12 @@ Or, to install with MongoDB support:

python3.7 -m pip install -U Red-DiscordBot[mongo]

Or, to install with PostgreSQL support:

.. code-block:: none

python3.7 -m pip install -U Red-DiscordBot[postgres]

.. note::

To install the development version, replace ``Red-DiscordBot`` in the above commands with the
@@ -65,7 +65,7 @@ Installing Red
If you're not inside an activated virtual environment, include the ``--user`` flag with all
``pip`` commands.

* No MongoDB support:
* Normal installation:

.. code-block:: none

@@ -77,6 +77,12 @@ Installing Red

python -m pip install -U Red-DiscordBot[mongo]

* With PostgreSQL support:

.. code-block:: none

python3.7 -m pip install -U Red-DiscordBot[postgres]

.. note::

To install the development version, replace ``Red-DiscordBot`` in the above commands with the
@@ -14,11 +14,11 @@ for /F "tokens=* USEBACKQ" %%A in (`git ls-files "*.py"`) do (
goto %1

:reformat
black -l 99 !PYFILES!
black -l 99 --target-version py37 !PYFILES!
exit /B %ERRORLEVEL%

:stylecheck
black -l 99 --check !PYFILES!
black -l 99 --check --target-version py37 !PYFILES!
exit /B %ERRORLEVEL%

:newenv
@@ -33,7 +33,7 @@
from redbot.core.cli import interactive_config, confirm, parse_cli_flags
from redbot.core.core_commands import Core
from redbot.core.dev_commands import Dev
from redbot.core import __version__, modlog, bank, data_manager
from redbot.core import __version__, modlog, bank, data_manager, drivers
from signal import SIGTERM


@@ -99,7 +99,11 @@ def main():
)
cli_flags.instance_name = "temporary_red"
data_manager.create_temp_config()
loop = asyncio.get_event_loop()

data_manager.load_basic_configuration(cli_flags.instance_name)
driver_cls = drivers.get_driver_class()
loop.run_until_complete(driver_cls.initialize(**data_manager.storage_details()))
redbot.logging.init_logging(
level=cli_flags.logging_level, location=data_manager.core_data_path() / "logs"
)
@@ -111,7 +115,6 @@ def main():
red = Red(
cli_flags=cli_flags, description=description, dm_help=None, fetch_offline_members=True
)
loop = asyncio.get_event_loop()
loop.run_until_complete(red.maybe_update_config())
init_global_checks(red)
init_events(red, cli_flags)
@@ -251,7 +251,7 @@ def __init__(self, bot: Red):
user: discord.Member,
points: Optional[int] = 1,
*,
reason: str
reason: str,
):
"""Warn the user for the specified reason.
@@ -1,18 +1,17 @@
import asyncio
import inspect
import os
import logging
import os
from collections import Counter
from enum import Enum
from importlib.machinery import ModuleSpec
from pathlib import Path
from typing import Optional, Union, List

import discord
import sys
from discord.ext.commands import when_mentioned_or

from . import Config, i18n, commands, errors
from . import Config, i18n, commands, errors, drivers
from .cog_manager import CogManager

from .rpc import RPCMixin
@@ -592,8 +591,8 @@ class Red(RedBase, discord.AutoShardedClient):

async def logout(self):
"""Logs out of Discord and closes all connections."""

await super().logout()
await drivers.get_driver_class().teardown()

async def shutdown(self, *, restart: bool = False):
"""Gracefully quit Red.

0 comments on commit d1a46ac

Please sign in to comment.
You can’t perform that action at this time.