Be able to iterate over an iterable/generator that might have errors

[thorwhalen]

Be able to iterate over an iterable/generator that might have errors.

Note the lined function

def consume_until_error(iterable, caught_errors=(Exception,)):
    """Iterable that will simply exit with out error if one of the caught
    errors occurs.
    >>> list(consume_until_error(map(lambda x: 1 / x, [4, 2, 1, 0, -1])))
    [0.25, 0.5, 1.0]
    """
    caught_errors = cast_to_tuple_if_non_iterable_or_a_string(caught_errors) + (
        StopIteration,
    )
    it = iter(iterable)
    while True:
        try:
            yield next(it)
        except caught_errors:
            break

Generalize this to a iterate_skipping_errors

def iterate_skipping_errors(
    g: Iterable,
    error_callback: Optional[Callable[[BaseException], Any]] = None,
    caught_exceptions: Tuple[BaseException] = (Exception,),
) -> Generator:
"""
    Iterate over a generator, skipping errors and calling an error callback if provided.

    >>> list(iterate_skipping_errors(map(lambda x: 1 / x, [1, 0, 2])))
    [1.0, 0.5]
    >>> list(iterate_skipping_errors(map(lambda x: 1 / x, [1, 0, 2]), print))
    division by zero
    [1.0, 0.5]
"""

function that has a error_callback that can be used to handle the errors in different ways, the default being just to ignore them silently.

This function can then be used with functools.partial to make a custom parametrized version.

[thorwhalen]

The iterate_skipping_errors() function can be useful when you want to iterate over a generator or iterable that may raise exceptions, but you don't want those exceptions to halt your program or script. Instead, you want to skip over the items that cause exceptions and continue iterating over the rest of the items.

The function takes an iterable g, an optional error callback function error_callback, and a tuple of exceptions caught_exceptions that should be caught and skipped. The function returns a generator that yields the values of the original generator, skipping over any items that raised a caught exception.

The function is especially useful when dealing with external data sources, such as web APIs, where there may be intermittent errors that you want to ignore and continue processing data as much as possible.

However, there are a few cons or warnings to consider when using this function:

If the iterable g raises a non-caught exception, such as a TypeError or ValueError, the function will still raise that exception and halt your program. You should make sure that the exceptions you specify in caught_exceptions cover all of the exceptions that could be raised by g.

Skipping errors without taking any action may result in incorrect or incomplete data processing. You should make sure that you have a way to recover from errors, such as retrying the operation, logging the error for later analysis, or providing default values for the skipped items.

The error callback function error_callback can be used to take some action when an error occurs, such as logging the error or printing a message. However, the function is called synchronously and may slow down your program or script if it performs expensive or time-consuming operations.

Overall, the iterate_skipping_errors() function can be a useful tool for processing data that may contain errors, but you should be aware of the potential risks and take appropriate measures to handle errors and recover from incomplete or incorrect data processing.