Refactor storage operations into separate Backend classes #348

elfjes · 2020-10-15T06:30:51Z

Following the discussion in #253 and #325 I've created a first iteration on what a Backend interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles

app.py talks only to core.py with regards to package operations
at configuration time, a Backend implementation is chosen and created for the lifetime of the configured app
core.py proxies requests for packages to this Backend()
The Backend interface/api is defined through three things
- methods that an implementation must implement
- methods that an implementation may override if it knows better than the defaults
- the PkgFIle class that is (should be) the main carrier of data
where possible, implementation details must be hidden from concrete Backends to promote extensibility

Other things I've done in this PR:

I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
Better testability of the CacheManager (no more race conditions when watchdog is installed during testing)
Cleanup some more Python 2 code
Started moving away from os.path and py.path in favour of pathlib

Furthermore I've created a plugin.py with a sample of how I think plugin system could look like. This sampIe assumes we use argparse and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.

The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs

Simplify the PgkFile class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of PkgFile so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
Add a PkgFile.metadata that backend implementations may use to store custom data for packages. For example the current PkgFile.root attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
Use pathlib wherever possible. This may also result in less attributes for PkgFile, since some things may be just contained in a single Path object, instead of multtiple strings.
Improve testing of the CacheManager.

…terface

mplanchard · 2020-10-15T12:54:58Z

Awesome! Just FYI I have what will probably be a long day of work ahead and some plans for this evening, so I probably won’t get to review this until Friday evening, but I’m looking forward to it!

…

On Thu, Oct 15, 2020 at 1:31 AM PelleK ***@***.***> wrote: Following the discussion in #253 <#253> and #325 <#325> I've created a first iteration on what a Backend interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles - app.py talks only to core.py with regards to package operations - at configuration time, a Backend implementation is chosen and created for the lifetime of the configured app - core.py proxies requests for packages to this Backend() - The Backend interface/api is defined through three things - methods that an implementation must implement - methods that an implementation may override if it knows better than the defaults - the PkgFIle class that is (should be) the main carrier of data - where possible, implementation details must be hidden from concrete Backends to promote extensibility Other things I've done in this PR: - I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means - Better testability of the CacheManager (no more race conditions when watchdog is installed during testing) - Cleanup some more Python 2 code - Started moving away from os.path and py.path in favour of pathlib Furthermore I've created a plugin.py with a sample of how I think plugin system could look like. This sampIe assumes we use argparse and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR. The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs - Simplify the PgkFile class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of PkgFile so that its only concern is being a data carrier between the app and the backends, and make its use more clear. - Add a PkgFile.metadata that backend implementations may use to store custom data for packages. For example the current PkgFile.root attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it. - Use pathlib wherever possible. This may also result in less attributes for PkgFile, since some things may be just contained in a single Path object, instead of multtiple strings. - Improve testing of the CacheManager. ------------------------------ You can view, comment on, or merge this pull request online at: #348 Commit Summary - move some functions around in preparation for backend module - rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil - further implement the current filestorage as simple file backend - rename prefix to project, since that's more descriptive - add digester func as attribute to pkgfile - WIP caching backend - WIP make cache better testable - better testability of cache - WIP file backends as plugin - Merge branch 'master' of github.com:elfjes/pypiserver into backend_interface File Changes - *M* pypiserver/__init__.py <https://github.com/pypiserver/pypiserver/pull/348/files#diff-eb39dcb6c5edb691cf2c59ec607eba8aaca08cc8c4bd5f49f6b4354f31ba5d73> (3) - *M* pypiserver/__main__.py <https://github.com/pypiserver/pypiserver/pull/348/files#diff-7e55e15c67273f0925ecd703161471f0ebc3562841787e4d4b336cfc950a7e6d> (4) - *M* pypiserver/_app.py <https://github.com/pypiserver/pypiserver/pull/348/files#diff-6485df1d3c02eb24b709ff35b4bce34060ecc909153df3b51c43c4a592693fed> (111) - *A* pypiserver/backend.py <https://github.com/pypiserver/pypiserver/pull/348/files#diff-4626005dbcd09690415b2d2f8a1b1535fab767ecb8da813f4402d7f0dce0380e> (224) - *M* pypiserver/cache.py <https://github.com/pypiserver/pypiserver/pull/348/files#diff-7a67bdfe8ae3dc5180bf7cd3b79be6bcf1801d0377c3c097e52f4550885c12c6> (42) - *M* pypiserver/core.py <https://github.com/pypiserver/pypiserver/pull/348/files#diff-dc372a5afa8581783c82548716c3539c6bdef35475464a6cd5d11e1037454ebf> (396) - *M* pypiserver/manage.py <https://github.com/pypiserver/pypiserver/pull/348/files#diff-fbd074c517ad1c416858c39cae72b25c8efbb8c16b2a01f0a9b127cc7a1f8152> (51) - *A* pypiserver/pkg_helpers.py <https://github.com/pypiserver/pypiserver/pull/348/files#diff-1894d0e278f38176f9be68ed996f81409c813b5fb917c0044d00ebeef9c10698> (107) - *A* pypiserver/plugin.py <https://github.com/pypiserver/pypiserver/pull/348/files#diff-032e81e4a9a172bc2858d71558fe76081f31c0b422375696a512a11a5ac1970b> (41) - *M* tests/test_app.py <https://github.com/pypiserver/pypiserver/pull/348/files#diff-a67cb1853203a6f1956991a9d9881d231c4d43557f5baffd45cc672a87e41cc6> (45) - *M* tests/test_core.py <https://github.com/pypiserver/pypiserver/pull/348/files#diff-938de46e643ab091cff6a7c23e4e752e8907e2266f47889d988923352f7a1058> (68) - *M* tests/test_main.py <https://github.com/pypiserver/pypiserver/pull/348/files#diff-41d180f6f7233d175c2dfe36c45c4ea80eed4754fa7ed4eed46ccde1b2a778c7> (8) - *M* tests/test_manage.py <https://github.com/pypiserver/pypiserver/pull/348/files#diff-7e8346b0ea680b9921d653a97988fed4c0d2b24fdb5eba7d9132d2a8e32dd3dd> (38) Patch Links: - https://github.com/pypiserver/pypiserver/pull/348.patch - https://github.com/pypiserver/pypiserver/pull/348.diff — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#348>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABTQRYL5PHNN7QHKFJKRHNLSK2JKXANCNFSM4SRRMEYQ> .

elfjes · 2020-10-15T16:14:16Z

Sure no worries. Looking forward to your feedback 😄

mplanchard · 2020-10-19T01:19:54Z

Got a chance to give this a quick overview this evening, and I really like the direction! I have an in-progress branch to replace all usage of the old config with the stuff from #339 with all tests passing. You can see that here. It seems like that could mesh really well with the approach taken here. I think we would update the RunConfig to expose a backend attribute, rather than having it expose an iter_packages method.

I've got no major concerns with what I've seen so far, but I'll be sure to make time to give this a more thorough review over the course of this week.

In regards to the things you mentioned as potentially needing more work:

Simplify the PgkFile class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of PkgFile so that its only concern is being a data carrier between the app and the backends, and make its use more clear.

I agree with this goal! I don't feel strongly either way on whether it should be a part of this PR, but I definitely think it's worth exploring

Add a PkgFile.metadata that backend implementations may use to store custom data for packages. For example the current PkgFile.root attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.

Yeah, and having some access to metadata will also give us a route towards e.g. fully supporting PEP 503 and the data_requires attribute (a la #320).

Use pathlib wherever possible. This may also result in less attributes for PkgFile, since some things may be just contained in a single Path object, instead of multtiple strings.

I've been working in this direction in my branch as well, so definitely no issues with seeing more of it.

Improve testing of the CacheManager.

Yeah the existing watchdog cache had really minimal tests, so anything here is an improvement

Regarding getting both of our streams of work going harmoniously, we can either get this merged, and then I can rebase and update on top of this, or vice-versa. I'll probably go ahead and open a PR with that branch tonight or tomorrow, just so that we can better look at how they might work together, but there's no rush to get that one in, and I'm happy to do the work of rebasing if we get this one in first.

mplanchard

Found some time this evening for a first pass!

mplanchard · 2020-10-19T02:18:16Z

pypiserver/core.py


+def with_digester(func: t.Callable[..., t.Iterable[PkgFile]]):


to keep the decorator from overriding the underlying type, you can use a generic like:

DigesterFunc = t.TypeVar("DigesterFunc", bound=t.Callable[..., t.Iterable[PkgFile]]) def with_digester(func: DigesterFunc) -> DigesterFunc: ...

you may need a # type: ignore where you're returning your wrapper function inside the decorator, but by telling mypy that you're returning a type of the same type that was put into the function, you allow it to infer that the resulting decorated function takes the same arguments as the original

good idea! :)

pypiserver/manage.py

mplanchard · 2020-10-19T02:22:22Z

tests/test_app.py

+
+# @pytest.mark.xfail(
+#     ENABLE_CACHING, reason="race condition when caching is enabled"
+# )


good catch, will remove

mplanchard · 2020-10-19T02:25:04Z

pypiserver/_app.py

@@ -104,7 +91,7 @@ def root():
    fp = request.custom_fullpath

    try:
-        numpkgs = len(list(packages()))
+        numpkgs = len(list(core.get_all_packages()))


It would be nice if we added a "package_count" or similar method to the backend so that, for example, we wouldn't actually have to parse all the files into PkgFile instances in order to get their count, and could instead just nab all files whose names match a regex or something. It would also be useful with imagined remote backends, where pulling all the files might be a problem

a package_count method sounds good, but I'm not sure about adding "query" or filter functionality. This would require additional complexity from each backend (including said imagined ones) without adding apparent value. What would be a use case for having such filtering functionality in a package_count method?

Sorry, I just meant a count without any custom filtering, but I expressed that poorly. Where I was talking about a regex, I essentially just meant "count every file that looks like a package based on the filename"

mplanchard · 2020-10-19T02:28:38Z

pypiserver/_app.py

    links = [
-        (f.relfn_unix, urljoin(fp, f.fname_and_hash(config.hash_algo)))
-        for f in files
+        (pkg.relfn_unix, urljoin(fp, pkg.fname_and_hash)) for pkg in packages
    ]


Could we try replacing this list comprehension ([]) with a generator comprehension (())? We definitely need to allocate a list for sorting, but here I think we may be able to be lazy and just pass a generator into the template call, which should save some RAM on systems handling a lot of packages

Yeah sure :)

~~Good catch on the fname_and_hash. Apparently no test caught that~~ nevermind

mplanchard · 2020-10-19T02:44:06Z

pypiserver/backend.py

+        )
+
+
+def as_file(fh: t.BinaryIO, destination: PathLike):


bikeshed: as_file() or to_file() feels to me like the function will be returning something file-like. What about write_file()?

mplanchard · 2020-10-19T02:46:11Z

pypiserver/backend.py

+        os.remove(pkg.fn)
+
+    def exists(self, filename):
+        # TODO: Also look in subdirectories?


I think currently we're recursing into root directories, so we may want to do so here too

mplanchard · 2020-10-19T02:47:31Z

pypiserver/backend.py

+
+class CachingFileBackend(SimpleFileBackend):
+    def __init__(
+        self, config: Configuration, roots: t.List[PathLike], cache_manager


We may want to define either a base class or a protocol for the cache_manager

Alas, no Protocols in python<3.8. I see that I put the type_hint for cache_manager in the __init__ assignment rather that the function signature, so I'm changing that.

Did you mean to add a sort of CacheManager interface/ base implementation that different backends may consume and/or extend? This might be an idea, but I'm afraid that there is not going to be much common requirements between the different backends, since most of the logic will lie in cache invalidation, which is going to be very backend-specific

Alas, no Protocols in python<3.8

This is true, but I think that if we want to use it, you can get it for earlier pythons by installing typing-extensions.

Did you mean to add a sort of CacheManager interface/ base implementation that different backends may consume and/or extend

Yep, that was the idea, so that you could potentially have like a WatchdogCachingFileBackend and a WhateverOtherCachingFileBackend, so long as the mangers share the same interface. It seems like all
we need for that at the moment is listdir() and digest_file(), so it might not be too bad to specify an abstract interface

mplanchard · 2020-10-19T02:48:16Z

pypiserver/backend.py

+        )
+
+
+class SimpleFileBackend(Backend):


I think mypy will probably want us to define types for the subclasses. Since this is a new file, we can add it to the typechecks in tox.ini to make sure it catches anything we miss

I'm not too familiar with mypy and I'm not quite sure what you mean with this. A subclass is a type, right? Or do you mean something else?
I'll add the new file(s) to the tox.ini and see what needs be done :)

Yep, I just meant that we would need to type these methods for mypy to be happy, and that mypy will enforce that they match the base class types to satisfy the liskov substitution principle

pypiserver/backend.py

elfjes · 2020-10-19T18:51:59Z

Thanks for the review, much appreciated :)

Got a chance to give this a quick overview this evening, and I really like the direction! I have an in-progress branch to replace all usage of the old config with the stuff from #339 with all tests passing. You can see that here. It seems like that could mesh really well with the approach taken here. I think we would update the RunConfig to expose a backend attribute, rather than having it expose an iter_packages method.

Yeah I was also thinking of having a --backend attribute which picks a backend. See also plugin.py in which I philosophized a bit on how we can support selection and configuration of different backends

Simplify the PgkFile class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of PkgFile so that its only concern is being a data carrier between the app and the backends, and make its use more clear.

I agree with this goal! I don't feel strongly either way on whether it should be a part of this PR, but I definitely think it's worth exploring

Maybe it's best to give it it's own PR, so that we don't have to make this one larger than it already is :)

Add a PkgFile.metadata that backend implementations may use to store custom data for packages. For example the current PkgFile.root attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.

Yeah, and having some access to metadata will also give us a route towards e.g. fully supporting PEP 503 and the data_requires attribute (a la #320).

Yeah I realized that there are two types of metadata that we must distinguish. One is the PEP503 metadata that we need for full support of this PEP. The other is geared towards extra data that a specific backend implementation may need to keep track of it's packages, such as a package root for a FileBackend, a bucket name for an imagined MultiBucketGCSBackend or a package's primary key in a database, and have a designated place to store such additional attributes. My thought initially was with the latter, but we need the former as well.

Regarding getting both of our streams of work going harmoniously, we can either get this merged, and then I can rebase and update on top of this, or vice-versa. I'll probably go ahead and open a PR with that branch tonight or tomorrow, just so that we can better look at how they might work together, but there's no rush to get that one in, and I'm happy to do the work of rebasing if we get this one in first.

Yeah sure, I'll finish up this PR and then create a new one for the additional work to ease the integration pain a little bit

Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>

@elfjes

This PR is a pretty substantial refactor of the entrypoints of pypiserver (`__main__` and `__init__`) to use the argparse-based config added in #339. - Updated `RunConfig` and `UpdateConfig` classes to have exclusive init kwargs, instead of taking an namespace. This turned out to be much easier when working with the library-style app initialization in `__init__`, both for direct instantiation and via paste config - Added an `iter_packages()` method to the `RunConfig` to iterate over packages specified by the configuration (note @elfjes, I think that replacing this with e.g. a `backend` reference will be a nice way to tie in #348) - Added a general-purpose method to map legacy keyword arguments to the `app()` and `paste_app_factory()` functions to updated forms - Refactored the `paste_app_factory()` to not mutate the incoming dictionary - Removed all argument-parsing and config-related code from `__main__` and `core` - Moved `_logwrite` from `__init__` to `__main__`, since that was the only place it was being used after the updates to `core` - Updated `digest_file` to use `hashlib.new(algo)` instead of `getattr(hashlib, algo)`, because the former supports more algorithms - Updated `setup.py` to, instead of calling `eval()` on the entirety of `__init__`, to instead just evaluate the line that defines the version - Assigned the config to a `._pypiserver_config` attribute on the `Bottle` instance to reduce hacky test workarounds - Fixed the tox config, which I broke in #339 * Config: add auth & absolute path resolution * Config: check pkg dirs on config creation * Instantiate config with kwargs, not namespace * WIP: still pulling the threads * Init seems to be working * tests passing locally, still need to update cache * Fix tox command * unused import * Fix typing * Be more selective in exec() in setup.py * Require accurate casing for hash algos * Remove old comment * Comments, minor updates and simplifications * move _logwrite to a more reasonable place * Update config to work with cache * Type cachemanager listdir in core * Update config module docstring, rename method * Add more comments re: paste config * Add comments to main, remove unneded check * Remove commented code * Use {posargs} instead of [] for clarity in tox * Add dupe check for kwarg updater * Remove unused references on app instance * Fix typo * Remove redundancy in log level parsing

…nd_interface

elfjes · 2020-10-30T15:31:15Z

I've added a --backend option to the cli that by default automatically chooses between the caching and the simple file backend.

It took me a while to figure out how to exactly add the argument and make it work with the Config.default_with_overrides function that is used in the tests. Which got me thinking, if we want to add pluggable backends, these new backends may want to add new command line arguments. How would we deal with them in the currently design of the (Run)Config, which has fixed supported arguments? I guess we'd have to go back to some form of dynamic config objects, although that's probably a discussion for a different issue/PR

Speaking of pluggable backends, @mplanchard when you have time, could you give me your feedback on the proposal I made for plugins in plugin.py?

Also, if you're ok with the current status of this PR, we can merge it, so that I can start working on the other things in a separate PR.

mplanchard · 2020-11-15T22:15:04Z

It took me a while to figure out how to exactly add the argument and make it work with the Config.default_with_overrides function that is used in the tests

There is definitely still room for improvement in making these obvious/easy to add to.

Which got me thinking, if we want to add pluggable backends, these new backends may want to add new command line arguments. How would we deal with them in the currently design of the (Run)Config, which has fixed supported arguments? I guess we'd have to go back to some form of dynamic config objects, although that's probably a discussion for a different issue/PR

I think the way we would approach this would be to have a dynamic portion of the config object, e.g. backend_config, which is a generic namespace. Plugins would need to provide some function to parse arguments and return a namespace, which we'd use to populate that property. Or something like that 🤷

elfjes · 2020-11-15T22:43:30Z

yeah that could work. Or like a **custom argument to the config object that gets converted to a namespace. Or just kept as a dictionary

mplanchard · 2020-11-15T22:15:35Z

.coveragerc

@@ -0,0 +1,2 @@
+[run]
+omit = pypiserver/bottle.py


good thinking

mplanchard · 2020-11-15T22:26:55Z

pypiserver/cache.py

+
+    Observer = None
+
+    ENABLE_CACHING = False


Not for this PR, but I wonder if we should (for 2.0) make this a config option instead of implicitly using the cache if the package is available. The implicitness feels potentially surprising to me.

Ah I see you added this already!

mplanchard · 2020-11-15T23:11:32Z

pypiserver/config.py

+
+        backend = available_backends[arg]
+
+        return BackendProxy(backend(self))


Should we put this in the _ConfigCommon so that it's available for the update command too?

Yes, that makes sense to me. I think the main reason I didn't do that yet, was because it currently requires hash_algo to be part of the same config and that one's on the RunConfig. But if you're fine with moving that to common, then I see no reason to not do that.

Yep, now that we need it in both the run and update configs, we might as well move it

mplanchard · 2020-11-15T23:13:42Z

pypiserver/config.py



 class _ConfigCommon:
+    hash_algo: t.Optional[str] = None


Should we move the hash_algo init argument to this common config?

mplanchard · 2020-11-15T23:14:21Z

pypiserver/core.py

+        "digest",  # The file digest in the form of <algo>=<hash>
+        "relfn_unix",  # The relative file path in unix notation
+        "parsed_version",  # The package version as a tuple of parts
+        "digester",  # a function that calculates the digest for the package


thanks for this documentation!

Haha, I mostly put it there for my own sanity until I have the chance to refactor PkgInfo

mplanchard · 2020-11-15T23:15:34Z

pypiserver/core.py

        self.fn = fn
        self.root = root
        self.relfn = relfn
        self.relfn_unix = None if relfn is None else relfn.replace("\\", "/")
        self.replaces = replaces
+        self.digest = None


Should this be digester to match the __slots__?

Both digest and digester are part of slots. I do see that I need to assign self.digester=None in __init__ to be consistent

mplanchard · 2020-11-15T23:17:30Z

pypiserver/plugin.py

+from pypiserver.backend import SimpleFileBackend, CachingFileBackend
+from pypiserver import get_file_backend
+
+DEFAULT_PACKAGE_DIRECTORIES = ["~/packages"]


Can we pull this from config.DEFAULTS?

So in my opinion, if we're doing this as plugins, then the package root is a config setting specific to the file backends. Which would also mean that the cli argument should not be part of the global config, but provided by the plugin. Which is why I added that definition there, and not take it from the global config

That does make sense. I just want to avoid needing to update this twice if we update it (although the likelihood that we'll ever change this is pretty low tbf)

…nd_interface

elfjes · 2021-01-15T08:58:09Z

Hi @mplanchard,

It's been a while, how are you? What do you think about the status of this PR? As far as I'm concerned it's ready to be merged. I've processed most, if not all of your feedback so far. Are there still any blocking things?

mplanchard

Sorry for being slow on this. Q4 and the beginning of Q1 have been crazy at my day job. Let's get it in!

mplanchard · 2021-02-02T03:39:04Z

pypiserver/config.py

+        # if arg not in available_backends:
+        #     raise argparse.ArgumentTypeError(
+        #         f"Value must be one of {', '.join(available_backends.keys())}"
+        #     )


Ah thanks, yeah I thought I might need to check that here, but it's actually picked up by argparse

mplanchard · 2021-02-02T03:41:33Z

Leaving it to you to merge. Let me know if you don't have permissions, and I'll hit the button instead

dee-me-tree-or-love · 2021-02-02T15:08:08Z

@mplanchard @elfjes if no rights to merge, I can do also!
(sorry for my low participation lately as well... also quite overloaded with various tasks on the job. @mplanchard, what do you also think, actually is it okay for me to also be merging/approving when I get a moment for it, maybe I will be poking for your call upfront in the future still?)

mplanchard · 2021-02-02T15:23:47Z

@mplanchard, what do you also think, actually is it okay for me to also be merging/approving when I get a moment for it, maybe I will be poking for your call upfront in the future still

Yeah I think so! For bigger stuff, we should probably both review, but for small things I'm okay with either of us approving and merging. And feel free to tag me for a review if there's anything at all you're uncertain about or would like me to weigh in on. If there's anything time sensitive, bug me in Zulip. It's harder for me to ignore real time chat than GH issues 🤪.

elfjes · 2021-02-02T17:28:46Z

I removed that dead code you mentioned, so I guess it needs reapproval. I don't have a merge button so if any of you @mplanchard @dee-me-tree-or-love can merge it, I'd be much obliged :)

elfjes added 10 commits October 10, 2020 16:06

move some functions around in preparation for backend module

0dac621

rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil

42f5e5a

further implement the current filestorage as simple file backend

c396cbc

rename prefix to project, since that's more descriptive

3e99067

add digester func as attribute to pkgfile

c959372

WIP caching backend

dd12820

WIP make cache better testable

b4e288f

better testability of cache

50a7796

WIP file backends as plugin

e95c8ad

Merge branch 'master' of github.com:elfjes/pypiserver into backend_in…

a162cd1

…terface

fix typos, run black

0e32bd4

mplanchard mentioned this pull request Oct 19, 2020

Use argparse config throughout app #349

Merged

mplanchard reviewed Oct 19, 2020

View reviewed changes

elfjes and others added 9 commits October 19, 2020 20:56

Apply suggestions from code review

aefa316

Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>

add more type hints to pass mypy, fix tox.ini

b0fa1fe

add package count method to backend

dee063a

add package count method to backend

559546c

minor changes

2af9649

bugfix when checking invalid whl file

176a717

check for existing package recursively, bugfix, some more pathlib

055d49f

fix unittest

666c5d6

rm dead code

abc1ae0

elfjes added 3 commits October 30, 2020 11:48

Merge branch 'master' of github.com:/pypiserver/pypiserver into backe…

a8ddaed

…nd_interface

exclude bottle.py from coverage

528736d

fix merge mistakes

f4728b3

elfjes added 2 commits October 30, 2020 12:19

fix tab indentation

4fe6238

backend as a cli argument

669b8b6

elfjes mentioned this pull request Oct 30, 2020

Add PypiserverApp app factory #350

Open

elfjes added 3 commits October 30, 2020 15:05

fix cli, add tests

d8961fe

fix mypy

46c6a48

fix more silly mistakes

52912a9

This was referenced Nov 5, 2020

Add support for data-requires-python #320

Open

WIP: AppEngine plugin as part of the package #327

Closed

Merge branch 'master' into backend_interface

70566b9

mplanchard reviewed Nov 15, 2020

View reviewed changes

elfjes added 2 commits December 7, 2020 11:42

Merge branch 'master' of github.com:/pypiserver/pypiserver into backe…

54c1b39

…nd_interface

process feedback

55ab56a

mplanchard previously approved these changes Feb 2, 2021

View reviewed changes

dee-me-tree-or-love mentioned this pull request Feb 2, 2021

On modernization, code quality and contributions. #325

Open

9 tasks

remove dead code

995b13e

elfjes dismissed mplanchard’s stale review via 995b13e February 2, 2021 16:53

mplanchard approved these changes Feb 2, 2021

View reviewed changes

mplanchard merged commit cf424c9 into pypiserver:master Feb 2, 2021


		def with_digester(func: t.Callable[..., t.Iterable[PkgFile]]):


		backend = available_backends[arg]

		return BackendProxy(backend(self))

Refactor storage operations into separate Backend classes #348

Refactor storage operations into separate Backend classes #348

Conversation

elfjes commented Oct 15, 2020

mplanchard commented Oct 15, 2020 via email

elfjes commented Oct 15, 2020

mplanchard commented Oct 19, 2020

mplanchard left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elfjes Oct 19, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elfjes commented Oct 19, 2020 • edited Loading

elfjes commented Oct 30, 2020

mplanchard commented Nov 15, 2020

elfjes commented Nov 15, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elfjes Nov 16, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elfjes commented Jan 15, 2021

mplanchard left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mplanchard commented Feb 2, 2021

dee-me-tree-or-love commented Feb 2, 2021

mplanchard commented Feb 2, 2021

elfjes commented Feb 2, 2021

elfjes Oct 19, 2020 •

edited

Loading

elfjes commented Oct 19, 2020 •

edited

Loading

elfjes Nov 16, 2020 •

edited

Loading