Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-3889] Make File Monitoring Function checkpointable. #1984

Closed
wants to merge 3 commits into from

Commits on May 11, 2016

  1. [FLINK-3896] Allow a StreamTask to be Externally Cancelled

    It adds a method failExternally() to the StreamTask, so that custom Operators
    can make their containing task fail when needed.
    kl0u committed May 11, 2016
    Configuration menu
    Copy the full SHA
    7deb922 View commit details
    Browse the repository at this point in the history
  2. [FLINK-3717] Make FileInputFormat checkpointable

    This adds a new interface called CheckpointableInputFormat
    which describes input formats whose state is queryable,
    i.e. getCurrentChannelState() returns where the reader is
    in the underlying source, and they can resume reading from
    a user-specified position.
    
    This functionality is not yet leveraged by current readers.
    kl0u committed May 11, 2016
    Configuration menu
    Copy the full SHA
    c9682b7 View commit details
    Browse the repository at this point in the history

Commits on May 18, 2016

  1. [FLINK-3889] Refactor File Monitoring Source

    This is meant to replace the different file
    reading sources in Flink streaming. Now there is
    one monitoring source with DOP 1 monitoring a
    directory and assigning input split to downstream
    readers.
    
    In addition, it makes the new features added by
    FLINK-3717 work together with the aforementioned entities
    (the monitor and the readers) in order to have
    fault tolerant file sources and exactly once guarantees.
    
    This does not replace the old API calls. This
    will be done in a future commit.
    kl0u committed May 18, 2016
    Configuration menu
    Copy the full SHA
    f5d95fc View commit details
    Browse the repository at this point in the history