New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[V3 Downloader] Revision tracking #2571
[V3 Downloader] Revision tracking #2571
Conversation
8a40137
to
e43271e
Compare
Okay, I think, this is ready for review now, I added all things from my TODO list (which you can find in previous revision of issue description) and I changed whole issue description and updated it with all changes this PR does. At the end, I have one question, should I make any new unit tests? I don't have a lot of experience with those, so I didn't add any, but if I should, I'll look into it. |
I've marked this for 3.2 due to breakage concerns. While we haven't explicitly exposed some of this, changes to downloader have a large potential for negative impact if we don't announce every potential change in behavior with advance warning. |
3112ca5
to
32f0559
Compare
- changes to `Repo.checkout()`: - `exit_to_rev` is now keyword only argument - added `force_checkout` to force checkout even if `Repo.commit` value is the same as target hash
- added two keyword arguments: - `valid_exit_codes` which specifies valid exit codes, used to determine if stderr should be sent as debug or error level in logging - `debug_only` which specifies if stderr can be sent only as debug level in logging
- fix wrong type annotations and add a lot of new ones - add checks for `Installable.repo` being `None` - fix wrong return type in `Downloader._install_requirements` - show repo names correctly when updating all repos - fix error when some requirement fails to install BREAKING CHANGE: - type of `Repo.available_modules` is now consistent (always `tuple`)
This script aims to help update the human-readable version of repo used for git integration tests in ``redbot/tests/downloader_testrepo.export`` by exporting/importing it in/from provided directory. Note ---- Editing `downloader_git_test_repo.export` file manually is strongly discouraged, especially editing any part of commit directives as that causes a change in the commit's hash. Another problem devs could encounter when trying to manually edit that file are editors that will use CRLF instead of LF for new line character(s) and therefore break it. I also used `.gitattributes` to prevent autocrlf from breaking testrepo. Also, if Git ever changes currently used SHA-1 to SHA-256 we will have to update old hashes with new ones. But it's a small drawback, when we can have human-readable version of repo. Known limitations ----------------- ``git fast-export`` exports commits without GPG signs so this script disables it in repo's config. This also means devs shouldn't use ``--gpg-sign`` flag in ``git commit`` within the test repo.
Also added Markdown file that is even more clear than export file on what the test repo contains. This is manually created but can be automated on later date.
These tests use expected output that is already guaranteed by git tests.
I know this doesn't actually give any benefit to people that don't have write permission to the repo but I saw other big fella devs doing this, so I think this might be advisable.
follow-up on Cog-Creators#2605, this commit fully fixes Cog-Creators#1866
…om `IntEnum` There's desync of `InstallableType` class types due to hot-reload and `IntEnum` allows for equality check between different types
Type
Description of the changes
This PR implements features proposed in #2527. (closes #2527)
Fully fixes #1866 - as said in #2605, the change was easier to do after this PR is finished so I decided to do it in this PR as well so that it actually gets done before 3.2 release.
Fixes #2927 - I added a check for installed cogs that have the same name and added a attribute to
InstalledModule
that keeps repo name as it was saved in Config.Due to size of the changes, I added myself to
CODEOWNERS
file.I know this doesn't actually give any benefit to people that don't have write permission to the repo but I saw other big fella devs doing this, so I assume this is what I should do.
Internal info: All function parameters are explicitly called
hash
, if it can only be commit's full sha1 hash orrev
if it can be anything that names a commit object, see linkDownloader cog - changed/added commands:
[p]repo update
updates repos without updating any cogs[p]cog install
can now install multiple cogs[p]cog installversion
allows you to install cogs from specific revision of repo[p]cog installversion <repo_name> <revision> <cogs>
[p]cog pin
and[p]cog unpin
allows you to pin and unpin cogs (block/unblock updates)[p]cog checkforupdates
checks for available cog updates (listing will include pinned cogs)[p]cog updateallfromrepos
updates all cogs from given repos[p]cog updatetoversion
updates chosen (or all) cogs to specific revision of repo[p]cog updatetoversion <repo_name> <revision [cogs]
Downloader's tests:
Here's somewhat big change, I added git integration tests that will make sure that all the git commands we use give expected output. Most git commands require a repo to operate on so along with this I added a test repo and a new (cool) script at
tools/edit_testrepo.py
that will allow devs to easily import and export that repo from git fast-import format. I'm not gonna list all of the new tests but other changes that were done indirectly to tests are:asyncio.subprocess
and default Windows loop doesn't work with it)pytest-mock
requirement totests
extra as mocking was very needed to tests Downloader's logic without depending on calling git part.Aaaaand, here you have changes that are rather boring for average reader:
Unfortunately, few public methods had to have their behaviour changed and therefore this PR is breaking change for devs using Downloader framework. To make it a little easier, I added [BREAKING] at the end of each change, which can be considered breaking (hopefully I haven't missed anything).
Downloader cog - internal changes:
installed_cogs = {'repo_name': {'cog_name': cog_json}}
_maybe_update_config()
installed_libraries()
gives you installed libraries andinstalled_modules()
gives you installed cogs and librariesRepoManager
andInstallable
classes - changes:RepoManager
now hasrepos
attribute which gives youtuple
ofRepo
RepoManager.get_all_cogs()
gets all cogs from all reposRepoManager.update_repo()
returns 2-tuple instead of dictionary with one key [BREAKING]Installable
holds commit information nowInstallable
now hasrepo
attribute containing repo object orNone
if repo ismissing
InstalledModule
class inheriting fromInstallable
- all installed cogs/libraries should be represented by thisfrom_installable()
method ofInstalledModule
creates instance fromInstallable
to_json()
andfrom_json()
methods moved fromInstallable
toInstalledModule
cog_name
key in internal cog's json changed its name tomodule_name
, new keyscommit
andpinned
(only for cogs) addedRepo
class - changes:Repo
holds commit information nowRepo.is_ancestor()
checks if one revision is ancestor of the other.Repo.is_on_branch()
checks if repo is currently on branchRepo.get_last_module_occurrence()
tries to get module from last commit in which it still occurs (if it couldn't, it returnsNone
)Repo.get_modified_modules()
gets modified modules between two revisions and if any module doesn't exist in new revision, it will try usingRepo.get_last_module_occurrence()
to find it.Repo.get_full_sha1()
gets full sha1 object name from given revision.Repo.checkout()
checks out repo to provided revision. The method can be awaited or used as async ctx manager:exit_to_rev
argument)rev
isNone
, method won't do anything and whenforce_checkout
is set toTrue
, checkout will be done even ifRepo.commit
equalsrev
/exit_to_rev
Repo.current_branch()
will now properly error out if repo is in detached HEAD state instead of returning "HEAD" [BREAKING]Repo.current_commit()
now returns commit, which the repo is on, instead of latest commit of branch,Repo.latest_commit()
provides old behavior of this method [BREAKING]Repo._get_file_update_statuses()
will now work properly with paths with spaces and with renames (treating them separately as addition and removal)Repo._run()
calls will redirect stderr to debug log, if valid exit code is returned by process or to error log otherwiseRepo.install_cog()
returnsInstalledModule
instance now instead of bool representing success of install [BREAKING]CopyingError
Repo.install_libraries()
returns 2-tuple of installed and failed libraries instead of bool representing the success of install [BREAKING]