Skip to content

Commit

Permalink
Change S3Proxy from memory to filesystem storage
Browse files Browse the repository at this point in the history
This should avoid OutOfMemoryErrors seen with larger object sizes.
Enabled by fixes in S3Proxy 2.0.0.  Also change LC_ALL to en_US.UTF-8
and add locales packages so that S3Proxy can create Unicode file
names.  This commit requires Java 17 on macOS to allow creating
extended attributes for directories.
  • Loading branch information
gaul committed Jun 24, 2023
1 parent 7c9cf84 commit 828658c
Show file tree
Hide file tree
Showing 3 changed files with 4 additions and 3 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ jobs:
- name: Install brew other packages
run: |
S3FS_BREW_PACKAGES='automake cppcheck python3 coreutils gnu-sed shellcheck';
S3FS_BREW_PACKAGES='automake cppcheck python3 coreutils gnu-sed openjdk@17 shellcheck';
for s3fs_brew_pkg in ${S3FS_BREW_PACKAGES}; do if brew list | grep -q ${s3fs_brew_pkg}; then if brew outdated | grep -q ${s3fs_brew_pkg}; then HOMEBREW_NO_AUTO_UPDATE=1 brew upgrade ${s3fs_brew_pkg}; fi; else HOMEBREW_NO_AUTO_UPDATE=1 brew install ${s3fs_brew_pkg}; fi; done;
- name: Install awscli2
Expand Down
2 changes: 1 addition & 1 deletion test/integration-test-common.sh
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ function start_s3proxy {
S3PROXY_CACERT_FILE=""
fi

"${STDBUF_BIN}" -oL -eL java -jar "${S3PROXY_BINARY}" --properties "${S3PROXY_CONFIG}" &
"${STDBUF_BIN}" -oL -eL java -Dfile.encoding=UTF8 -jar "${S3PROXY_BINARY}" --properties "${S3PROXY_CONFIG}" &
S3PROXY_PID=$!

# wait for S3Proxy to start
Expand Down
3 changes: 2 additions & 1 deletion test/s3proxy.conf
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ s3proxy.credential=local-credential
s3proxy.keystore-path=/tmp/keystore.jks
s3proxy.keystore-password=password

jclouds.provider=transient
jclouds.provider=filesystem
jclouds.identity=remote-identity
jclouds.credential=remote-credential
jclouds.filesystem.basedir=/var/tmp/blobstore

0 comments on commit 828658c

Please sign in to comment.