New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
I don't know what's happening during update_feeds() #204
Comments
The simplest solution (to implement) is making an update_feeds_iter() that yields (feed URL, exception or none) tuples. update_feeds() could just call that (in a way, it already does). This could also allow us to bubble up _NotModified (since it may be useful to users).
Update: What about warnings? Update: Should we also return the (new, updated) counts? |
Here's the whole update data flow as of v1.13.
|
We have two possible ways of using update_feeds_iter(): # a progressbar(item_show_func=...) will show the last completed item
with click.progressbar(update_feeds_iter()) as bar:
for result in bar:
# item is already done
pass
# a progressbar(item_show_func=...) will show the item that was last started;
# in practice, it will be identical to the snippet above due to
# https://github.com/pallets/click/issues/1353
with click.progressbar(update_feeds_iter()) as bar:
for url, process in bar:
# the item is not done yet
result = process()
# now it's done The first version is straightforward to implement. The second one is not:
# much pseudo-code
def update_start(feed):
nonlocal multi
# start a sub-bar with one item (either not done or done)
multi.add(feed.url, Progress(label=feed.url, length=1))
with MultiProgress(update_feeds_iter(update_start=update_started)) as multi:
for result in multi:
# finishes the bar corresponding to url
multi.update(result.url, 1) Given that the last example is essentially the first version + update_start (i.e. we can add update_start later), I'll just start with the first version. |
So, to decide what update_feed_iter() should return, we'll work backwards from what we'd like the output of the
The minimum thing to be returned for that would be something like: @dataclass(frozen=True)
class UpdateResult:
url: str
# updated/new and exception are mutually-exclusive;
# maybe "not modified" is signaled by updated/new is None AND exception is None
updated: Optional[int]
new: Optional[int]
exception: Optional[ParseError] Alternatively, we could model the mutual-exclusiveness from the start: @dataclass(frozen=True)
class UpdateResultOK:
url: str
updated: int
new: int
@dataclass(frozen=True)
class UpdateResult:
url: str
value: Optional[UpdateResultOK, ParseError]
# we can still have the attributes from the first version as properties
# (i.e. we can go with that first, and extend to this later)
@property
def updated(self) -> Optional[int]:
return self.value.updated if isinstance(self.value, UpdateOK) else None
... To either of these, we can later add Also, to show how many feeds are left, get_feed_counts() needs to support the same filter as update_feeds(), i.e. |
The final version of the types. Getting the count of feeds to be updated will happen in #217. class UpdatedFeed:
url: str
updated: int
new: int
class UpdateResult(NamedTuple):
url: str
value: Union[UpdatedFeed, None, ReaderError]
# the exception type is ReaderError and not ParseError
# to allow suppressing new errors without breaking the API:
# adding a new type to the union breaks the API,
# not raising an exception type anymore doesn't.
# currently, storage or plugin-raised exceptions
# prevent updates for the following feeds,
# but that's not necessarily by design.
# convenience methods, to show the semantics; optional
@property
def error(self) -> Optional[ReaderError]:
return self.value if isinstance(self.value, Exception) else None
@property
def ok(self) -> Optional[UpdatedFeed]:
return self.value if not isinstance(self.value, Exception) else None
@property
def not_modified(self) -> bool:
return self.value is None
def update_feeds_iter(...) -> Iterable[UpdateResult]:
# raises StorageError
...
def update_feeds(...) -> None:
# raises StorageError
...
# def update_feed(...) -> None:
def update_feed(...) -> Optional[UpdatedFeed]:
# raises FeedNotFoundError
# raises ParseError
# raises StorageError
... |
Remaining:
|
Time spent:
The implementation bit includes the docstrings; the documentation bit only means the user guide and changelog. The refactoring part should have taken only 1 hour, but I made things more complicated than they should have been: instead of removing NotModified completely from the start, I made it internal to Parser and wasted time handling it (81da965, dd7a1a3). |
I don't know what's happening during update_feeds() until it finishes. More so, there's no straightforward way to know programmatically which feeds failed (logging or guessing from feed.last_exception don't count).
From https://clig.dev/#robustness-guidelines:
Doing either of these is hard to do at the moment.
The text was updated successfully, but these errors were encountered: