-
Notifications
You must be signed in to change notification settings - Fork 1.4k
This document covers the setup of S3 in Overleaf Community Edition and Server Pro. A separate document can be found on migrating existing data to S3.
For instances with fewer than 1000 seats we recommend using local disk storage with regular backups. For larger instances with more than 1000 seats that reach limits of their local storage (size or throughput), we recommend using a S3 compatible object storage back end over other network based storage solutions like NFS.
Here are the most popular options for S3 compatible object storage:
- AWS S3, managed, we suggest picking AWS S3 when running Server CE/Pro on AWS
- Min.io, self-hosted
- Ceph.io, self-hosted
- Other hosting providers also have some kind of managed S3 compatible object storage, you may want to use these instead of running your own when already running Server CE/Pro at such a provider.
The latency between the Server CE/Pro instance and your S3 compatible object storage is a big contributor to the time it takes to complete the migration. The latency also impacts the file-upload performance in Server CE/Pro and slow file-downloads can have a big impact on PDF compile times as well. We suggest minimizing the geo-graphical distance between your Server CE/Pro instance and the S3 compatible object storage. In a managed environment, this would mean provisioning a bucket in the same region, and for an on-premise solution, running the two on the same campus.
We need four "buckets" and two restricted user accounts.
The buckets should not be publicly accessible.
Note: The history related items are needed from Server CE/Pro v3.5 onwards.
Bucket | Usage | Service | Previously in /var/lib/sharelatex/data
|
---|---|---|---|
overleaf-user-files |
project user files | filestore | user_files |
overleaf-template-files |
template files | filestore | template_files |
overleaf-project-blobs |
project history blobs | history | history/overleaf-project-blobs |
overleaf-chunks |
history chunks | history | history/overleaf-chunks |
You may want/need to pick a different name, be sure to use the custom buckets in all the commands.
The following will use placeholders for actual credentials:
-
SHARELATEX_FILESTORE_S3_ACCESS_KEY_ID
This is the access key/username of the restricted user of the filestore service. -
SHARELATEX_FILESTORE_S3_SECRET_ACCESS_KEY
This is the secret key/password of the restricted user of the filestore service. -
SHARELATEX_HISTORY_S3_ACCESS_KEY_ID
This is the access key/username of the restricted user of the history service. -
SHARELATEX_HISTORY_S3_SECRET_ACCESS_KEY
This is the secret key/password of the restricted user of the history service.
Server CE/Pro only needs a small set of permissions on each bucket:
- create object
- get object
- delete object
- list bucket
Here is how a policy for the filestore user could look like:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::overleaf-user-files"
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::overleaf-user-files/*"
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::overleaf-template-files"
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::overleaf-template-files/*"
}
]
}
Here is how a policy for the history user could look like:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::overleaf-project-blobs"
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::overleaf-project-blobs/*"
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::overleaf-chunks"
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::overleaf-chunks/*"
}
]
}
# Enable S3 backend for filestore
SHARELATEX_FILESTORE_BACKEND=s3
# Bucket name for project files
SHARELATEX_FILESTORE_USER_FILES_BUCKET_NAME=overleaf-user-files
# Bucket name for template files
SHARELATEX_FILESTORE_TEMPLATE_FILES_BUCKET_NAME=overleaf-template-files
# Key for filestore user
SHARELATEX_FILESTORE_S3_ACCESS_KEY_ID=...
# Secret for filestore user
SHARELATEX_FILESTORE_S3_SECRET_ACCESS_KEY=...
# Bucket region you picked when creating the buckets.
SHARELATEX_FILESTORE_S3_REGION=""
# Enable S3 backend for history
SHARELATEX_HISTORY_BACKEND=s3
# Bucket name for project history blobs
SHARELATEX_HISTORY_PROJECT_BLOBS_BUCKET=overleaf-project-blobs
# Bucket name for history chunks
SHARELATEX_HISTORY_CHUNKS_BUCKET=overleaf-chunks
# Key for history user
SHARELATEX_HISTORY_S3_ACCESS_KEY_ID=...
# Secret for history user
SHARELATEX_HISTORY_S3_SECRET_ACCESS_KEY=...
# Bucket region you picked when creating the buckets.
SHARELATEX_HISTORY_S3_REGION=""
# Enable S3 backend for filestore
SHARELATEX_FILESTORE_BACKEND=s3
# Bucket name for project files
SHARELATEX_FILESTORE_USER_FILES_BUCKET_NAME=overleaf-user-files
# Bucket name for template files
SHARELATEX_FILESTORE_TEMPLATE_FILES_BUCKET_NAME=overleaf-template-files
# Key for filestore user
SHARELATEX_FILESTORE_S3_ACCESS_KEY_ID=...
# Secret for filestore user
SHARELATEX_FILESTORE_S3_SECRET_ACCESS_KEY=...
# S3 provider endpoint
SHARELATEX_FILESTORE_S3_ENDPOINT=http://10.10.10.10:9000
# Path style addressing of buckets. Most likely you need to set this to "true".
SHARELATEX_FILESTORE_S3_PATH_STYLE="true"
# Bucket region. Most likely you do not need to configure this.
SHARELATEX_FILESTORE_S3_REGION=""
# Enable S3 backend for history
SHARELATEX_HISTORY_BACKEND=s3
# Bucket name for project history blobs
SHARELATEX_HISTORY_PROJECT_BLOBS_BUCKET=overleaf-project-blobs
# Bucket name for history chunks
SHARELATEX_HISTORY_CHUNKS_BUCKET=overleaf-chunks
# Key for history user
SHARELATEX_HISTORY_S3_ACCESS_KEY_ID=...
# Secret for history user
SHARELATEX_HISTORY_S3_SECRET_ACCESS_KEY=...
# S3 provider endpoint
SHARELATEX_HISTORY_S3_ENDPOINT=http://10.10.10.10:9000
# Path style addressing of buckets. Most likely you need to set this to "true".
SHARELATEX_HISTORY_S3_PATH_STYLE="true"
# Bucket region. Most likely you do not need to configure this.
SHARELATEX_HISTORY_S3_REGION=""
Note: MINIO_ROOT_USER
and MINIO_ROOT_PASSWORD
are the root credentials of
the minio instance.
Please follow the official documentation for obtaining a copy of mc
.
mc alias set s3 http://10.10.10.10:9000 MINIO_ROOT_USER MINIO_ROOT_PASSWORD
# Put the contents of the policies from the previous section in the
# respective json file policy-filestore.json and policy-history.json.
# Reminder: Replace the bucket names and credentials accordingly.
# filestore buckets, user and policy
mc mb --ignore-existing s3/overleaf-user-files
mc mb --ignore-existing s3/overleaf-template-files
mc admin user add s3 \
SHARELATEX_FILESTORE_S3_ACCESS_KEY_ID \
SHARELATEX_FILESTORE_S3_SECRET_ACCESS_KEY
mc admin policy create s3 overleaf-filestore policy-filestore.json
mc admin policy attach s3 overleaf-filestore \
--user=SHARELATEX_FILESTORE_S3_ACCESS_KEY_ID
# history buckets, user and policy
mc mb --ignore-existing s3/overleaf-project-blobs
mc mb --ignore-existing s3/overleaf-chunks
mc admin user add s3 \
SHARELATEX_HISTORY_S3_ACCESS_KEY_ID \
SHARELATEX_HISTORY_S3_SECRET_ACCESS_KEY
mc admin policy create s3 overleaf-history policy-history.json
mc admin policy attach s3 overleaf-history \
--user=SHARELATEX_HISTORY_S3_ACCESS_KEY_ID
- Quickstart Guide (Overleaf Toolkit)
- Hardware Requirements
- Database & Dependencies
- Creating and managing users
- General configuration
- Configuring Email
- SSL & Nginx reverse proxy
- Data and Backups
- Configuring Headers, Footers & Logo
- Password Restrictions
- i18n Languages
- Logging
- Common Config Options
- F.A.Q
- Troubleshooting
- Full Project History Migration