Skip to content
Browse files

[V3 Downloader] Revision tracking (#2571)

* feat(downloader): Install cog from specific commit in repo (initial commit)

- Repo and Installable have commit property now
- New class inheriting from Installable -
InstalledCog (old one from removed)
- New Repo.checkout() method, which is also async ctx manager
ref #2527

* fix(downloader): Keep information about repo's branch in config

- This is needed to make sure that repo can go back from detached state in some rare unexpected
- current branch is determined by `git symbolic-ref` now as this command errors for detached

* feat(downloader): Update repo without cogs, update single cog

The most important part of issue #2527 has been added here
- `[p]repo update` command added
- new conf format - nested dictionary repo_name->cog_name->cog_json
  installed libraries are now kept in conf too
  - `InstalledCog` renamed to `InstalledModule` - installed libraries use this class
  - `Downloader.installed_libraries()` and `Downloader.installed_modules()` added
  - `Downloader._add_to_installed()` and `Downloader._remove_from_installed()`
    now accept list of modules, of both cogs and libraries
- `[p]cog install` tells about fails of copying cog and installing shared libraries
- `[p]cog update` will truly update only chosen cogs (if provided) or cogs that need update
  - pinned cogs aren't checked
  - before update, repos are updated
  - to determine if update is needed `Repo.get_modified_modules()` is used
- `[p]cog pin` and `[p]cog unpin` commands for pinning/unpinning cogs added
- `Repo.checkout()` allows to choose ctx manager exit's checkout revision
- `Repo.install_cog()` returns `InstalledModule` now and raises CopyingError (maybe breaking?)
- `Repo.install_libraries()` returns 2-tuple of installed and failed libraries (maybe breaking?)
- `RepoManager.get_all_cogs()` added, which returns cogs from all repos
- `RepoManager.repos` property added, which contains tuple of `Repo`

* test(downloader): Repo.current_branch() throws an exception, when branch can't be determined

* style(downloader): rename _add_to_installed to _save_to_installed

This method is used for both adding and updating existing modules in Config

* refactor(downloader): add ctx.typing() for few commands

`[p]cog install` is nested hell, can't wait for moving install logic to separate method

* fix(downloader): refactor and fix `set` usage

* perf(downloader): update commits for ALL checked modules to omit diffs next time

This will also disable running git diff for cogs that have the same commit as the latest one

* style(downloader): few style improvements

- use of mutable object in method definition
- make Repo._get_full_sha1() public method
- too long
- don't use len to check if sequence is empty

* feat(downloader): add `[p]cog updateallfromrepos` and `[p]cog updatetoversion` commands

- moved cog update logic into `Downloader._cog_update_logic()` (lack of better name)
  - splitted
whole cog update process into smaller methods
  - might still need some improvements
- added new
methods to `Repo` class:
  - `is_on_branch()` to check if repo is currently checked out to branch

- `is_ancestor()` to check if one commit is ancestor of the other
- fix for
`Downloader._available_updates()` behaviour
broken by commit

* feat(downloader): try to find last commit where module is still present

- `Installable` now has `repo` attribute containing repo object or `None` if repo is
- `Downloader._install_cogs()` and `Downloader._reinstall_libraries()` are able to install
modules from different commits of repo
- `Repo.checkout()` as ctx manager will now exit to commit
which was active before checking out
- unification of `rev` and `hash` terms:
All function
parameters are explicitly called `hash`, if it can only be commit's full sha1 hash or `rev` if it
can be anything that names a commit object, see
- new
`Repo.get_last_module_occurence()` method, which gets module's Installable from last commit in which
it still occurs

* docs(downloader): Add basic description for `InstalledModule`

* fix(downloader): cog ignored during updates if its commit was missing

After config format update, commit string is empty until update and when such cog was checked and it
wasn't available in repo anymore, it was ignored

* refactor(downloader): Installing cogs from specific rev will pin them

* perf(downloader): Don't checkout when current commit equals target hash

- changes to `Repo.checkout()`:
  - `exit_to_rev` is now keyword only argument
  - added
`force_checkout` to force checkout even if `Repo.commit` value is the same as target hash

* refactor(downloader): Repo._run() stderr is redirected to debug log now

- added two keyword arguments:
  - `valid_exit_codes` which specifies valid exit codes, used to
determine if stderr should be sent as debug or error level in logging
  - `debug_only` which
specifies if stderr can be sent only as debug level in logging

* style(downloader): stop using `set` as arg name in `_load_repos()`

* feat(downloader): pass multiple cogs to `[p]cog (un)pin`

* refactor(downloader): accept module name instead of instance, fix spelling

* style(downloader): few small style changes

* fix(downloader): add type annotations + fixes based on them

- fix wrong type annotations and add a lot of new ones
- add checks for `Installable.repo` being `None`
- fix wrong return type in `Downloader._install_requirements`
- show repo names correctly when updating all repos
- fix error when some requirement fails to install

- type of `Repo.available_modules` is now consistent (always `tuple`)

* tests: use same event loop policy as in Red's code

* enhance(downloader): fully handle ambiguous revisions

* build(deps): add pytest-mock dependency to tests extra

* fix(downloader): minor fixes

* feat(downloader): add tool for editing Downloader's test repo

This script aims to help update the human-readable version of repo
used for git integration tests in ``redbot/tests/downloader_testrepo.export``
by exporting/importing it in/from provided directory.

Editing `downloader_git_test_repo.export` file manually is strongly discouraged,
especially editing any part of commit directives as that causes a change in the commit's hash.
Another problem devs could encounter when trying to manually edit that file
are editors that will use CRLF instead of LF for new line character(s) and therefore break it.

I also used `.gitattributes` to prevent autocrlf from breaking testrepo.

Also, if Git ever changes currently used SHA-1 to SHA-256 we will have to
update old hashes with new ones. But it's a small drawback,
when we can have human-readable version of repo.

Known limitations
``git fast-export`` exports commits without GPG signs so this script disables it in repo's config.
This also means devs shouldn't use ``--gpg-sign`` flag in ``git commit`` within the test repo.

* tests(downloader): add git tests and test repo for them

Also added Markdown file that is even more clear than export file
on what the test repo contains.
This is manually created but can be automated on later date.

* test(downloader): add more tests related to RepoManager

These tests use expected output that is already guaranteed by git tests.

* chore(CODEOWNERS): add jack1142 to Downloader's folders

I know this doesn't actually give any benefit to people that don't have
write permission to the repo but I saw other big fella devs doing this,
so I think this might be advisable.

* enhance(downloader): allow easy schema updates in future

* enhance(downloader): more typing fixes, add comments for clarity

* feat(downloader): add python and bot version check to update process

follow-up on #2605, this commit fully fixes #1866

* chore(changelog): add towncrier entries

* fix(downloader): use `*args` instead of `commands.Greedy`

* fix(downloader): hot-reload issue - `InstallableType` now inherits from `IntEnum`

There's desync of `InstallableType` class types due to hot-reload
and `IntEnum` allows for equality check between different types

* enhance(downloader): ensure there's no cog with same name installed

should fix #2927

* fix(downloader): last few changes before marking as ready for review
  • Loading branch information...
jack1142 authored and mikeshardmind committed Nov 8, 2019
1 parent d85fb26 commit e2c8b1100870511aaa575ce807f9009e5a5dba99
Showing with 2,892 additions and 422 deletions.
  1. +4 −1 .github/CODEOWNERS
  2. +1 −0 changelog.d/2571.misc.rst
  3. +1 −0 changelog.d/downloader/1866.enhance.rst
  4. +1 −0 changelog.d/downloader/
  5. +1 −0 changelog.d/downloader/2527.enhance.1.rst
  6. +1 −0 changelog.d/downloader/2527.enhance.2.rst
  7. +1 −0 changelog.d/downloader/2527.feature.1.rst
  8. +1 −0 changelog.d/downloader/2527.feature.2.rst
  9. +1 −0 changelog.d/downloader/2527.feature.3.rst
  10. +1 −0 changelog.d/downloader/2527.feature.4.rst
  11. +1 −0 changelog.d/downloader/2527.feature.5.rst
  12. +1 −0 changelog.d/downloader/2527.feature.6.rst
  13. +4 −0 changelog.d/downloader/2527.misc.1.rst
  14. +1 −0 changelog.d/downloader/2571.bugfix.1.rst
  15. +1 −0 changelog.d/downloader/2571.bugfix.2.rst
  16. +1 −0 changelog.d/downloader/2571.dep.rst
  17. +1 −0 changelog.d/downloader/2571.enhance.rst
  18. +1 −0 changelog.d/downloader/2571.misc.rst
  19. +1 −0 changelog.d/downloader/2927.bugfix.rst
  20. +6 −0 docs/framework_downloader.rst
  21. +21 −2 redbot/
  22. +3 −11 redbot/
  23. +1 −1 redbot/cogs/downloader/
  24. +3 −3 redbot/cogs/downloader/
  25. +824 −228 redbot/cogs/downloader/
  26. +38 −0 redbot/cogs/downloader/
  27. +85 −24 redbot/cogs/downloader/
  28. +7 −6 redbot/cogs/downloader/
  29. +576 −105 redbot/cogs/downloader/
  30. +1 −0 redbot/pytest/.gitattributes
  31. +117 −17 redbot/pytest/
  32. +134 −0 redbot/pytest/downloader_testrepo.export
  33. +102 −0 redbot/pytest/
  34. +2 −0 setup.cfg
  35. +315 −20 tests/cogs/downloader/
  36. +452 −0 tests/cogs/downloader/
  37. +4 −4 tests/cogs/downloader/
  38. +3 −0 tests/
  39. +172 −0 tools/
  40. +1 −0 tools/primary_deps.ini
@@ -33,7 +33,7 @@ redbot/cogs/audio/* @aikaterna
redbot/cogs/bank/* @tekulvw
redbot/cogs/cleanup/* @palmtree5
redbot/cogs/customcom/* @palmtree5
redbot/cogs/downloader/* @tekulvw
redbot/cogs/downloader/* @tekulvw @jack1142
redbot/cogs/economy/* @palmtree5
redbot/cogs/filter/* @palmtree5
redbot/cogs/general/* @palmtree5
@@ -49,6 +49,9 @@ redbot/cogs/warnings/* @palmtree5
# Docs
docs/* @tekulvw @palmtree5

# Tests
tests/cogs/downloader/* @jack1142

# Setup, instance setup, and running the bot @tekulvw
redbot/ @tekulvw
@@ -0,0 +1 @@
Tests now use same event loop policy as Red's code.
@@ -0,0 +1 @@
Downloader will now check if Python and bot version match requirements in ``info.json`` during update.
@@ -0,0 +1 @@
Added :func:`redbot.cogs.downloader.repo_manager.InstalledModule` to Downloader's framework docs.
@@ -0,0 +1 @@
User can now pass multiple cog names to ``[p]cog install``.
@@ -0,0 +1 @@
When passing cogs to ``[p]cog update`` command, it will now only update those cogs, not all cogs from the repo these cogs are from.
@@ -0,0 +1 @@
Added ``[p]repo update [repos]`` command that allows you to update repos without updating cogs from them.
@@ -0,0 +1 @@
Added ``[p]cog installversion <repo_name> <revision> <cogs>`` command that allows you to install cogs from specified revision (commit, tag, branch) of given repo. When using this command, cog will automatically be pinned.
@@ -0,0 +1 @@
Added ``[p]cog pin <cogs>`` and ``[p]cog unpin <cogs>`` for pinning cogs. Cogs that are pinned will not be updated when using update commands.
@@ -0,0 +1 @@
Added ``[p]cog checkforupdates`` command that will tell which cogs can be updated (including pinned cog) without updating them.
@@ -0,0 +1 @@
Added ``[p]cog updateallfromrepos <repos>`` command that will update all cogs from given repos.
@@ -0,0 +1 @@
Added ``[p]cog updatetoversion <repo_name> <revision> [cogs]`` command that updates all cogs or ones of user's choosing to chosen revision of given repo.
@@ -0,0 +1,4 @@
Added :func:`redbot.cogs.downloader.installable.InstalledModule` which is used instead of :func:`redbot.cogs.downloader.installable.Installable` when we refer to installed cog or shared library.
- ``to_json`` and ``from_json`` methods were moved from :func:`redbot.cogs.downloader.installable.Installable` to :func:`redbot.cogs.downloader.installable.InstalledModule`
- return types changed for :func:`redbot.cogs.converter.InstalledCog.convert`, :func:`redbot.cogs.downloader.Downloader.installed_cogs`, :func:`redbot.cogs.downloader.Repo.install_cog` to use :func:`redbot.cogs.downloader.installable.InstalledModule`.
@@ -0,0 +1 @@
Made regex for repo names use raw string to stop ``DeprecationWarning`` about invalid escape sequence.
@@ -0,0 +1 @@
Downloader will no longer allow to install cog that is already installed.
@@ -0,0 +1 @@
Added ``pytest-mock`` requirement to ``tests`` extra.
@@ -0,0 +1 @@
Added error messages for failures during installing/reinstalling requirements and copying cogs and shared libraries.
@@ -0,0 +1 @@
Added more Downloader tests for Repo logic and git integration. New git tests use a test repo file that can be generated using new tool at ``tools/``.
@@ -0,0 +1 @@
Downloader will no longer allow to install cog with same name as other that is installed.
@@ -68,6 +68,12 @@ Installable
.. autoclass:: Installable


.. autoclass:: InstalledModule

.. automodule:: redbot.cogs.downloader.repo_manager

@@ -1,3 +1,4 @@
import asyncio as _asyncio
import re as _re
import sys as _sys
import warnings as _warnings
@@ -15,8 +16,13 @@


__all__ = ["MIN_PYTHON_VERSION", "__version__", "version_info", "VersionInfo"]

__all__ = [
if _sys.version_info < MIN_PYTHON_VERSION:
f"Python {'.'.join(map(str, MIN_PYTHON_VERSION))} is required to run Red, but you have "
@@ -173,6 +179,19 @@ def __repr__(self) -> str:

def _update_event_loop_policy():
if _sys.platform == "win32":
elif == "cpython":
# Let's not force this dependency, uvloop is much faster on cpython
import uvloop as _uvloop
except ImportError:

__version__ = "3.1.7"
version_info = VersionInfo.from_str(__version__)

@@ -13,17 +13,9 @@
# Set the event loop policies here so any subsequent `get_event_loop()`
# calls, in particular those as a result of the following imports,
# return the correct loop object.
if sys.platform == "win32":
elif == "cpython":
# Let's not force this dependency, uvloop is much faster on cpython
import uvloop
except ImportError:
uvloop = None
from redbot import _update_event_loop_policy


import redbot.logging
from import Red, ExitCodes
@@ -21,7 +21,7 @@
_ = T_

async def do_install_agreement(ctx: commands.Context):
async def do_install_agreement(ctx: commands.Context) -> bool:
downloader = ctx.cog
if downloader is None or downloader.already_agreed:
return True
@@ -1,14 +1,14 @@
import discord
from redbot.core import commands
from redbot.core.i18n import Translator
from .installable import Installable
from .installable import InstalledModule

_ = Translator("Koala", __file__)

class InstalledCog(Installable):
class InstalledCog(InstalledModule):
async def convert(cls, ctx: commands.Context, arg: str) -> Installable:
async def convert(cls, ctx: commands.Context, arg: str) -> InstalledModule:
downloader ="Downloader")
if downloader is None:
raise commands.CommandError(_("No Downloader cog found."))

0 comments on commit e2c8b11

Please sign in to comment.
You can’t perform that action at this time.