Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[V3 Downloader] Revision tracking #2571

Merged
merged 37 commits into from Nov 8, 2019
Merged
Changes from 1 commit
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
e819548
feat(downloader): Install cog from specific commit in repo (initial c…
jack1142 Mar 24, 2019
d7cc3ad
fix(downloader): Keep information about repo's branch in config
jack1142 Mar 29, 2019
b4220c4
feat(downloader): Update repo without cogs, update single cog
jack1142 Apr 9, 2019
f5bef98
test(downloader): Repo.current_branch() throws an exception, when bra…
jack1142 Apr 9, 2019
fe9c218
style(downloader): rename _add_to_installed to _save_to_installed
jack1142 Apr 10, 2019
b6a8eb8
refactor(downloader): add ctx.typing() for few commands
jack1142 Apr 10, 2019
f867e3b
fix(downloader): refactor and fix `set` usage
jack1142 Apr 11, 2019
cfe7985
perf(downloader): update commits for ALL checked modules to omit diff…
jack1142 Apr 11, 2019
e36d90e
style(downloader): few style improvements
jack1142 Apr 13, 2019
7743630
feat(downloader): add `[p]cog updateallfromrepos` and `[p]cog updatet…
jack1142 Apr 17, 2019
4620558
feat(downloader): try to find last commit where module is still present
jack1142 Apr 19, 2019
79c95cb
docs(downloader): Add basic description for `InstalledModule`
jack1142 Apr 19, 2019
6df6c07
fix(downloader): cog ignored during updates if its commit was missing
jack1142 Apr 21, 2019
3f5f436
refactor(downloader): Installing cogs from specific rev will pin them
jack1142 Apr 21, 2019
2591413
perf(downloader): Don't checkout when current commit equals target hash
jack1142 Apr 21, 2019
e37629f
refactor(downloader): Repo._run() stderr is redirected to debug log now
jack1142 Apr 21, 2019
5601665
style(downloader): stop using `set` as arg name in `_load_repos()`
jack1142 Apr 21, 2019
a7eb010
feat(downloader): pass multiple cogs to `[p]cog (un)pin`
jack1142 Apr 21, 2019
18db287
refactor(downloader): accept module name instead of instance, fix spe…
jack1142 Apr 21, 2019
07fcebd
style(downloader): few small style changes
jack1142 Apr 21, 2019
997f247
fix(downloader): add type annotations + fixes based on them
jack1142 Jul 7, 2019
321298c
tests: use same event loop policy as in Red's code
jack1142 Oct 25, 2019
9ece0c8
enhance(downloader): fully handle ambiguous revisions
jack1142 Oct 25, 2019
5676df4
build(deps): add pytest-mock dependency to tests extra
jack1142 Oct 25, 2019
bd239fe
fix(downloader): minor fixes
jack1142 Oct 25, 2019
6c57736
feat(downloader): add tool for editing Downloader's test repo
jack1142 Oct 26, 2019
7b17d32
tests(downloader): add git tests and test repo for them
jack1142 Oct 26, 2019
32312b6
test(downloader): add more tests related to RepoManager
jack1142 Oct 25, 2019
cea4693
chore(CODEOWNERS): add jack1142 to Downloader's folders
jack1142 Oct 25, 2019
0821e12
enhance(downloader): allow easy schema updates in future
jack1142 Oct 25, 2019
029a621
enhance(downloader): more typing fixes, add comments for clarity
jack1142 Oct 25, 2019
da068f4
feat(downloader): add python and bot version check to update process
jack1142 Oct 25, 2019
ead2bce
chore(changelog): add towncrier entries
jack1142 Oct 26, 2019
30eaf6e
fix(downloader): use `*args` instead of `commands.Greedy`
jack1142 Oct 26, 2019
671f28c
fix(downloader): hot-reload issue - `InstallableType` now inherits fr…
jack1142 Oct 26, 2019
aaed251
enhance(downloader): ensure there's no cog with same name installed
jack1142 Oct 26, 2019
9fa5e6c
fix(downloader): last few changes before marking as ready for review
jack1142 Oct 27, 2019
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

tests(downloader): add git tests and test repo for them

Also added Markdown file that is even more clear than export file
on what the test repo contains.
This is manually created but can be automated on later date.
  • Loading branch information
jack1142 committed Oct 26, 2019
commit 7b17d32874d2cff3b6bbd2e399511106a0d40448
@@ -1,6 +1,8 @@
from collections import namedtuple
from pathlib import Path
import json
import subprocess as sp
import shutil

import pytest

@@ -20,6 +22,10 @@
"library_installable",
"fake_run_noprint",
"fake_latest_commit",
"_session_git_repo",
"git_repo",
"cloned_git_repo",
"git_repo_with_remote",
]


@@ -146,3 +152,93 @@ def library_installable(tmpdir):

cog_info = Installable(Path(str(lib_path)))
return cog_info


# Git
TEST_REPO_EXPORT_PTH: Path = Path(__file__).parent / "downloader_testrepo.export"


def _init_test_repo(destination: Path):
# copied from tools/edit_testrepo.py
git_dirparams = ("git", "-C", str(destination))
init_commands = (
(*git_dirparams, "init"),
(*git_dirparams, "config", "--local", "user.name", "Cog-Creators"),
(*git_dirparams, "config", "--local", "user.email", "cog-creators@example.org"),
(*git_dirparams, "config", "--local", "commit.gpgSign", "false"),
)

for args in init_commands:
sp.run(args, check=True)
return git_dirparams


@pytest.fixture(scope="session")
async def _session_git_repo(tmp_path_factory, event_loop):
# we will import repo only once once per session and duplicate the repo folder
repo_path = tmp_path_factory.mktemp("session_git_repo")
repo = Repo(
name="redbot-testrepo",
url="",
branch="master",
commit="",
folder_path=repo_path,
loop=event_loop,
)
git_dirparams = _init_test_repo(repo_path)
fast_import = sp.Popen((*git_dirparams, "fast-import", "--quiet"), stdin=sp.PIPE)
with TEST_REPO_EXPORT_PTH.open(mode="rb") as f:
fast_import.communicate(f.read())
return_code = fast_import.wait()
if return_code:
raise Exception(f"git fast-import failed with code {return_code}")
sp.run((*git_dirparams, "reset", "--hard"))
return repo


@pytest.fixture
async def git_repo(_session_git_repo, tmp_path, event_loop):
# fixture only copies repo that was imported in _session_git_repo
repo_path = tmp_path / "redbot-testrepo"
shutil.copytree(_session_git_repo.folder_path, repo_path)
repo = Repo(
name="redbot-testrepo",
url=_session_git_repo.url,
branch=_session_git_repo.branch,
commit=_session_git_repo.commit,
folder_path=repo_path,
loop=event_loop,
)
return repo


@pytest.fixture
async def cloned_git_repo(_session_git_repo, tmp_path, event_loop):
# don't use this if you want to edit origin repo
repo_path = tmp_path / "redbot-cloned_testrepo"
repo = Repo(
name="redbot-testrepo",
url=str(_session_git_repo.folder_path),
branch=_session_git_repo.branch,
commit=_session_git_repo.commit,
folder_path=repo_path,
loop=event_loop,
)
sp.run(("git", "clone", str(_session_git_repo.folder_path), str(repo_path)), check=True)
return repo


@pytest.fixture
async def git_repo_with_remote(git_repo, tmp_path, event_loop):
# this can safely be used when you want to do changes to origin repo
repo_path = tmp_path / "redbot-testrepo_with_remote"
repo = Repo(
name="redbot-testrepo",
url=str(git_repo.folder_path),
branch=git_repo.branch,
commit=git_repo.commit,
folder_path=repo_path,
loop=event_loop,
)
sp.run(("git", "clone", str(git_repo.folder_path), str(repo_path)), check=True)
return repo
@@ -0,0 +1,134 @@
# THIS FILE SHOULDN'T BE EDITED MANUALLY. USE `edit_testrepo.py` TOOL TO UPDATE THE REPO.
blob
mark :1
original-oid cfd75093008a560c1f2a09e5068e0dd1517eaa1c
data 14
Sample file 1.
reset refs/heads/ambiguous_with_tag
commit refs/heads/ambiguous_with_tag
mark :2
original-oid c6f0e5ec04d99bdf8c6c78ff20d66d286eecb3ea
author Cog-Creators <cog-creators@example.org> 1571921830 +0200
committer Cog-Creators <cog-creators@example.org> 1571919491 +0200
data 27
Commit ambiguous with tag.
M 100644 :1 sample_file1.txt

reset refs/heads/ambiguous_1
commit refs/heads/ambiguous_1
mark :3
original-oid 95da0b576271cb5bee5f3e075074c03ee05fed05
author Cog-Creators <cog-creators@example.org> 1571777704 +0200
committer Cog-Creators <cog-creators@example.org> 1571777704 +0200
data 23
Ambiguous commit 16955
M 100644 :1 sample_file1.txt

reset refs/heads/ambiguous_2
commit refs/heads/ambiguous_2
mark :4
original-oid 95da0b57a416d9c8ce950554228d1fc195c30b43
author Cog-Creators <cog-creators@example.org> 1571777704 +0200
committer Cog-Creators <cog-creators@example.org> 1571777704 +0200
data 23
Ambiguous commit 44414
M 100644 :1 sample_file1.txt

blob
mark :5
original-oid f1a18139c84a82addbded8a7b5738c36fb02fce1
data 22
print("Hello world!")

blob
mark :6
original-oid 1abb7a2470722faee2175980ee202717b4158057
data 14
Sample file 2.
reset refs/tags/lightweight
commit refs/tags/lightweight
mark :7
original-oid c950fc05a540dd76b944719c2a3302da2e2f3090
author Cog-Creators <cog-creators@example.org> 1571776887 +0200
committer Cog-Creators <cog-creators@example.org> 1571777047 +0200
data 31
Initial commit, prepare files.
M 100644 :5 mycog/__init__.py
M 100644 :1 sample_file1.txt
M 100644 :6 sample_file2.txt

blob
mark :8
original-oid 10ec5813415b6d7c902eee95cc13dc38c6f50917
data 11
Added file.
blob
mark :9
original-oid 5ed17bf7914989db85f2e66045e62b35eed10f3b
data 42
def setup(bot):
print("Hello world!")

commit refs/tags/lightweight
mark :10
original-oid fb99eb7d2d5bed514efc98fe6686b368f8425745
author Cog-Creators <cog-creators@example.org> 1571777140 +0200
committer Cog-Creators <cog-creators@example.org> 1571777140 +0200
data 39
Add, modify, rename and remove a file.
from :7
M 100644 :8 added_file.txt
M 100644 :9 mycog/__init__.py
D sample_file1.txt
D sample_file2.txt
M 100644 :6 sample_file3.txt

commit refs/tags/annotated
mark :11
original-oid a7120330cc179396914e0d6af80cfa282adc124b
author Cog-Creators <cog-creators@example.org> 1571777209 +0200
committer Cog-Creators <cog-creators@example.org> 1571777209 +0200
data 14
Remove mycog.
from :10
D mycog/__init__.py

blob
mark :12
original-oid 1ba9a868ae2f65571c75681ec47d40595bea4882
data 14
Sample file 4.
commit refs/heads/master
mark :13
original-oid 2db662c1d341b1db7d225ccc1af4019ba5228c70
author Cog-Creators <cog-creators@example.org> 1571777704 +0200
committer Cog-Creators <cog-creators@example.org> 1571777704 +0200
data 32
One commit after mycog removal.
from :11
M 100644 :12 sample_file4.txt

reset refs/heads/dont_add_commits
commit refs/heads/dont_add_commits
mark :14
original-oid a0ccc2390883c85a361f5a90c72e1b07958939fa
author Cog-Creators <cog-creators@example.org> 1571777548 +0200
committer Cog-Creators <cog-creators@example.org> 1571777548 +0200
data 103
Don't edit this, this is used for tests for current commit, latest commit, full sha1 from branch name.
M 100644 :1 sample_file1.txt

tag annotated
from :11
original-oid 41f6cf3b58e774d2b3414ced3ee9f2541f1c682f
tagger Cog-Creators <cog-creators@example.org> 1571777367 +0200
data 15
Annotated tag.

tag ambiguous_tag_66387
from :2
original-oid c6f028f843389c850e2c20d8dd1f5fa498252764
tagger Cog-Creators <cog-creators@example.org> 1571919491 +0200
data 37
Annotated tag ambiguous with commit.

@@ -0,0 +1,102 @@
# Downloader's test repo reference

This file can be used as a reference on what repo contains
if some dev will want to add more test in future.

Branch master
---

**Commit:** c950fc05a540dd76b944719c2a3302da2e2f3090
**Commit message:** Initial commit, prepare files.
**Tree status:**
```
downloader_testrepo/
├── mycog
A │ ├── __init__.py
A ├── sample_file1.txt
A └── sample_file2.txt
```
---
**Commit:** fb99eb7d2d5bed514efc98fe6686b368f8425745
**Tag:** lightweight
**Commit message:** Add, modify, rename and remove a file.
**Tree status:**
```
downloader_testrepo/
├── mycog/
M │ ├── __init__.py
A ├── added_file.txt
D ├── sample_file1.txt
R └── sample_file2.txt -> sample_file3.txt
```
---
**Commit:** a7120330cc179396914e0d6af80cfa282adc124b
**Tag:** annotated (sha1: 41f6cf3b58e774d2b3414ced3ee9f2541f1c682f)
**Commit message:** Remove mycog.
**Tree status:**
```
downloader_testrepo/
D ├── mycog/
D │ ├── __init__.py
├── added_file.txt
└── sample_file3.txt
```
---
**Commit:** 2db662c1d341b1db7d225ccc1af4019ba5228c70
**Commit message:** One commit after mycog removal.
**Tree status:**
```
downloader_testrepo/
├── added_file.txt
├── sample_file3.txt
A └── sample_file4.txt
```

Branch with persistent HEAD
---

**Commit:** a0ccc2390883c85a361f5a90c72e1b07958939fa
**Branch:** dont_add_commits
**Commit message:** Don't edit this, this is used for tests for current commit, latest commit, full sha1 from branch name.
**Tree status:**
```
downloader_testrepo/
A └── sample_file1.txt
```

Branches with ambiguous commits (95da0b57)
---

**Commit:** 95da0b576271cb5bee5f3e075074c03ee05fed05
**Branch:** ambiguous_1
**Commit message:** Ambiguous commit 16955
**Tree status:**
```
downloader_testrepo/
A └── sample_file1.txt
```


**Commit:** 95da0b57a416d9c8ce950554228d1fc195c30b43
**Branch:** ambiguous_2
**Commit message:** Ambiguous commit 44414
**Tree status:**
```
downloader_testrepo/
A └── sample_file1.txt
```


Branch with ambiguous tag (c6f0)
---

**Commit:** c6f0e5ec04d99bdf8c6c78ff20d66d286eecb3ea
**Branch:** ambiguous_with_tag
**Tag:** ambiguous_tag_66387 (sha1: c6f028f843389c850e2c20d8dd1f5fa498252764)
**Commit message:** Commit ambiguous with tag.
**Tree status:**

```
downloader_testrepo/
A └── sample_file1.txt
```
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.