Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FLUME-3217 . Creates empty files when quota #225

Open
wants to merge 11 commits into
base: trunk
Choose a base branch
from

Commits on Aug 17, 2018

  1. FLUME-3217 . Creates empty files when quota

    As it can be seen in FLUME-3217, Flume creates empty files when HDFS quota has been reached.
    
    This is a first approach to the solution, it works although a better approach it could be implemented. The idea is to capture the quota Exception in order to delete this annoying empty files generated by flume.
    ffernandez92 committed Aug 17, 2018
    Configuration menu
    Copy the full SHA
    da08bed View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2018

  1. Rollback AbstractHDFSWriter

    Avoid the deletion of empty files when quota reached as it could delete a file that does not have to be deleted.
    ffernandez92 committed Aug 19, 2018
    Configuration menu
    Copy the full SHA
    6adaafd View commit details
    Browse the repository at this point in the history
  2. FLUME-3217 - Update BucketWriter

    Adding the checkQuota function:
    -	When the quota is reached avoids creating new files as interrupts the open flow
    -	If the quota is reached just in the middle of a transaction, the file is closed with the data that could be included and then any other file will be created.
    ffernandez92 committed Aug 19, 2018
    Configuration menu
    Copy the full SHA
    3200b95 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8d0322a View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    db0dfbd View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    c173898 View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2018

  1. Configuration menu
    Copy the full SHA
    9d5bb3a View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2018

  1. FLUME-3217 . Checking FS type

    Check FS type before doing any action.
    ffernandez92 committed Aug 21, 2018
    Configuration menu
    Copy the full SHA
    a19a155 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2018

  1. FLUME-3217 . Inclusion of Replication Factor

    There was a mistake when the replication factor was greater than 1. I expect to commit a test shortly
    ffernandez92 committed Aug 30, 2018
    Configuration menu
    Copy the full SHA
    bfd4a82 View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2018

  1. Configuration menu
    Copy the full SHA
    264da73 View commit details
    Browse the repository at this point in the history
  2. FLUME-3217. Add HDFS Quota test

    FIX DEPENDENCIES.
    ffernandez92 committed Aug 31, 2018
    Configuration menu
    Copy the full SHA
    245862a View commit details
    Browse the repository at this point in the history