Using object storage

Using S3 compatible Object Storage

Encore supports using S3-compatible object storage for both input and output files. S3 URLs should be specified in the format s3://<BUCKET>/<KEY>.

For details on submitting jobs with S3 inputs and outputs, see the API and Job Submission guide and the Input Configuration guide.

Input from S3

When an S3 URL is used as input, encore will create a presigned URL that is then passed to ffmpeg for reading the file. Note that presigned URLs will be shown in the logs by default. If this is not desirable, you can configure encore to use a log configuration that masks the presign query parameters (see Configuration section).

Output to S3

When an S3 URL is used for outputFolder, the transcoding output will first be stored locally in a temporary directory, and then uploaded to S3 once transcoding is finished.

Authentication

AWS credentials are read using the AWS DefaultCredentialsProvider. This means credentials can be provided in multiple ways, including:

• Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
• AWS credentials file (~/.aws/credentials)
• IAM role (when running on EC2, ECS, or other AWS services)
• EC2 instance metadata service

For more details, see the AWS SDK documentation.

S3 URLs can also be accessed in anonymous mode, which corresponds to using the --no-sign-request flag with the AWS CLI. When anonymous access is enabled, any configured S3 access key or secret key will be ignored and multipart upload will be disabled (as the S3 SDK does not support multipart upload with anonymous access).

Multipart Upload

Encore uses the AWS CRT-based S3 client for efficient multipart uploads. The multipart upload behavior can be configured to optimize performance based on your network and file sizes. For detailed descriptions of these properties, see the AWS SDK documentation.

Configuration

S3 support must be explicitly enabled. All S3-related configuration properties are listed below:

Property	Env Variable	Type	Default	Description
remote-files.s3.enabled	REMOTEFILES_S3_ENABLED	Boolean	false	Enable S3 support
remote-files.s3.anonymousAccess	REMOTEFILES_S3_ANONYMOUSACCESS	Boolean	false	Enable anonymous access (no credentials required)
remote-files.s3.endpoint	REMOTEFILES_S3_ENDPOINT	String	-	Custom S3 endpoint for S3-compatible services (e.g., MinIO)
remote-files.s3.usePathStyle	REMOTEFILES_S3_USEPATHSTYLE	Boolean	true	Use path-style access (useful for S3-compatible services)
remote-files.s3.presignDurationSeconds	REMOTEFILES_S3_PRESIGNDURATIONSECONDS	Long	43200 (12 hours)	Duration of presigned URLs for input files
remote-files.s3.uploadTimeoutSeconds	REMOTEFILES_S3_UPLOADTIMEOUTSECONDS	Long	3600 (1 hour)	Timeout for upload operations
remote-files.s3.multipart.minimumPartSize	REMOTEFILES_S3_MULTIPART_MINIMUMPARTSIZE	Long (bytes)	Auto	Minimum size of each part in a multipart upload
remote-files.s3.multipart.threshold	REMOTEFILES_S3_MULTIPART_THRESHOLD	Long (bytes)	Auto	File size threshold for using multipart upload
remote-files.s3.multipart.targetThroughputGbps	REMOTEFILES_S3_MULTIPART_TARGETTHROUGHPUTGBPS	Double	Auto	Target throughput in Gbps
remote-files.s3.multipart.maxConcurrency	REMOTEFILES_S3_MULTIPART_MAXCONCURRENCY	Integer	Auto	Maximum number of concurrent upload operations
logging.config	LOGGING_CONFIG	String	-	Set to classpath:logback-json-mask-s3-presign.xml to mask presigned URLs in logs

Region: The AWS region is determined from the aws.region system property or AWS_REGION environment variable (defaults to us-east-1 if neither is set).

Credentials: See the Authentication section above for details on how credentials are provided.

Example YAML configuration:

remote-files:
  s3:
    enabled: true
    presignDurationSeconds: 43200
    uploadTimeoutSeconds: 3600
    multipart:
      minimumPartSize: 8388608
      threshold: 8388608
      targetThroughputGbps: 10.0
      maxConcurrency: 50

Example job with s3 input/output

{
  "profile": "x264-animated",
  "outputFolder": "s3://my-output-bucket/transcoded/",
  "baseName": "my-video",
  "inputs": [
    {
      "uri": "s3://my-input-bucket/sources/input-video.mp4",
      "type": "AudioVideo"
    }
  ]
}

The transcoded files will be uploaded to s3://my-output-bucket/transcoded/ with filenames based on the baseName and encoding profile settings.