Skip to content
Branch: master
Commits on Jun 14, 2019
  1. Fix filters not processing when there is no input.

    dwsteele committed Jun 14, 2019
    Some filters (e.g. encryption and compression) produce output even if there is no input.  Since the filter group was marked as "done" initially, processing would not run when there was zero input and that resulted in zero output.
    All filters start not done so start the filter group the same way.
Commits on Jun 13, 2019
  1. Use retries to wait for test S3 server to start.

    dwsteele committed Jun 13, 2019
    The prior method of tailing the docker log no longer seems reliable.  Instead, keep retrying the make bucket command until it works and show the error if it times out.
  2. Create filter group in read/write object's memory context.

    dwsteele committed Jun 13, 2019
    The filter group was being created in the parent context so it would go away if the parent context was freed.
  3. Remove extra spaces.

    dwsteele committed Jun 13, 2019
Commits on Jun 11, 2019
  1. Integrate S3 storage driver with HTTP client cache.

    dwsteele committed Jun 11, 2019
    This allows copying from one S3 object to another.  We generally try to avoid doing this but there are a few cases where it is needed and the tests do it quite a bit.
    One thing to look out for here is that reads require the http client to be explicitly released by calling httpClientDone().  This means than clients could grow if they are not released properly.  The http statistics will hopefully alert us if this is happening.
  2. Add HTTP client cache.

    dwsteele committed Jun 11, 2019
    This cache manages multiple http clients and returns one to the caller that is not busy.  It is the responsibility of the caller to indicate when they are done with a client. If returnContent is set then the client will automatically be marked done.
    Also add special handing for HEAD requests to recognize that content-length is informational only and no content is expected.
  3. Add tag to specify minio version to use for documentation build.

    dwsteele committed Jun 11, 2019
    The new minio major release broke the build.  We'll need to figure that out but for now use the last major version, which is known to work.
Commits on Jun 10, 2019
  1. Add info() and infoList() to S3 driver.

    dwsteele committed Jun 10, 2019
    These should be the last functions required to complete the implementation of the S3 driver.
  2. Revert removal of aws cli in 3e1b06a.

    dwsteele committed Jun 10, 2019
    This is required for integration tests to create buckets.
  3. Create general-purpose S3 list function.

    dwsteele committed Jun 10, 2019
    There are already two functions which need file lists and more on the way, so create a general-purpose function to reduce code duplication.
  4. Add missing const qualifiers.

    dwsteele committed Jun 10, 2019
Commits on Jun 7, 2019
  1. Return IoRead/IoWrite from io*FilterGroupSet()

    dwsteele committed Jun 7, 2019
    This makes is possible to chain functions more effectively.
Commits on Jun 5, 2019
  1. Expose storage type.

    dwsteele committed Jun 5, 2019
    Code calling the storage should be driver agnostic but knowing the storage type can be useful for debugging and error reporting.
  2. Enforce requiring repo-cipher-pass at config parse time.

    dwsteele committed Jun 5, 2019
    This was not enforced at parse time because repo1-cipher-type could be passed on the command-line even in cases where encryption was not needed by the subprocess.
    Filter repo-cipher-type so it is never passed on the command line.  If the subprocess does not have access to the passphrase then knowing the encryption type is useless anyway.
  3. Use wal_level=replica in the documentation for PostgreSQL >= 9.6.

    dwsteele committed Jun 5, 2019
    The documentation was using wal_level=hot_standby which is a deprecated setting.
    Also remove the reference to wal_level=archive since it is no longer supported and is not recommended for older versions.
    Suggested by Patrick McLaughlin.
Commits on Jun 4, 2019
  1. Add String to types supported by JSON list render.

    dwsteele committed Jun 4, 2019
    All types should be supported eventually but for now this answers the need.
  2. Fix incorrect error type on missing path.

    dwsteele committed Jun 4, 2019
  3. Make working with filter groups less restrictive.

    dwsteele committed Jun 4, 2019
    Filter groups could not be manipulated once they had been assigned to an IO object.  Now they can be freely manipulated up to the time the IO object is opened.
    Also, move the filter group into the IO object's context so they don't need to be tracked separately.
  4. Remove per-stanza repo cache clear during testing.

    dwsteele committed Jun 4, 2019
    This was not being used and is not supported by the equivalent C code.
  5. Use HEAD to check if a file exists on S3.

    dwsteele committed Jun 4, 2019
    The previous implementation searched for the file in a list which worked but was not optimal.  For arbitrary bucket structures it would also produce a false negative if a match was not found in the first 1000 entries.  This was not an issue for our repo structure since the max hits on exists calls is two but it seems worth fixing to avoid future complications.
  6. Make C S3 requests use the same host logic as Perl.

    dwsteele committed Jun 4, 2019
    The C code was passing the host (if specified) with the request which could force the server into path-style URLs, which are not supported.
    Instead, use the Perl logic of always passing bucket.endpoint in the request no matter what host is used for the HTTPS connection.
    It's an open question whether we should support path-style URLs but since we don't it's useless to tell the server otherwise.  Note that Amazon S3 has deprecated path-style URLs and they are no longer supported on newly created buckets.
  7. Fix erroneous parameter name.

    dwsteele committed Jun 4, 2019
    The exists() function only checks for files so the parameter name should reflect that.
Commits on Jun 1, 2019
  1. Bring back PATH_PAX for platforms that don't define it.

    dwsteele committed Jun 1, 2019
    This define was replaced in 8c712d8 with limits.h but that caused an issue with the hurd-i386 build for Debian which apparently does not define this value.
  2. Fix mismatched log/test macros.

    dwsteele committed Jun 1, 2019
  3. Add CPPFLAGS to compile rules.

    dwsteele committed Jun 1, 2019
    This should silence the last of the Debian package warnings.
Commits on May 31, 2019
  1. Fix build.flags being removed on each build.

    dwsteele committed May 31, 2019
    This was being removed by rsync which forced a full build even when a partial should have been fine.  Rewrite the file after the rsync so it is preserved.
  2. Remove Debian package patch merged to upstream.

    dwsteele committed May 31, 2019
Commits on May 30, 2019
  1. Fix typo in macro name.

    dwsteele committed May 30, 2019
  2. Add code classification exclusion missed in 3e1b06a.

    dwsteele committed May 30, 2019
Commits on May 29, 2019
  1. Build all docs with S3 using --var=s3-all=y

    dwsteele committed May 29, 2019
    Force repo-type=s3 for all tests.  This is not currently the default for any OS builds.
Commits on May 28, 2019
  1. Add internal flag for commands.

    dwsteele committed May 28, 2019
    Allow commands to be skipped by default in the command help but still work if help is requested for the command directly.  There may be other uses for the flag in the future.
    Update help for ls now that it is exposed.
  2. Add ls command.

    dwsteele committed May 28, 2019
    Allows listing repo paths/files from the command-line, to be used primarily for testing and debugging.
    This command is internal-only so the interface may change at any time without notice.
  3. Move tls/http statistics output to command/command.

    dwsteele committed May 28, 2019
    This module already has the filtering required to keep these messages from being displayed by default for commands that output to stdout (e.g. info).
Commits on May 27, 2019
  1. Use minio as local S3 emulator in documentation.

    dwsteele committed May 27, 2019
    The documentation was relying on a ScalityS3 container built for testing which wasn't very transparent.  Instead, use the stock minio container and configure it in the documentation.
    Also, install certificates and CA so that TLS verification can be enabled.
Commits on May 26, 2019
  1. Refactoring path support in the storage module.

    dwsteele committed May 26, 2019
    Not all storage types support paths as a physical thing that must be created/destroyed.  Add a feature to determine which drivers use paths and simplify the driver API as much as possible given that knowledge and by implementing as much path logic as possible in the Storage object.
    Remove the ignoreMissing parameter from pathSync() since it is not used and makes little sense.
    Create a standard list of error messages for the drivers to use and apply them where the code was modified -- there is plenty of work still to be done here.
You can’t perform that action at this time.