- Amazon S3 is an object store (flat-structure), meaning it is a higher layer data storage system, essentially it is a database "blob" storage, storing data in an underlying simple database as an object.
- It's designed for WRITE once READ many access i.e. append-only data structure, perfect for Static content.
- It's distributed and offers a very high level of redundancy i.e. data is stored in 3 AZs within a specific region.
| Use Case | Description |
|---|---|
| Static websites | You can configure your S3 bucket to host a static website of HTML, CSS, and client-side scripts. |
| Static content | Because of the limitless scaling, the support for large files, and the fact that you access any object over the web at any time, Amazon S3 is the perfect place to store static content. |
| Backup and storage | Amazon S3 is a natural place to back up files (example RDS Snapshots, EBS snapshots etc.) because it is highly redundant. |
| Media hosting | Because you can store unlimited objects, and each individual object can be up to 5 TBs, Amazon S3 is an ideal location to host video, photo, and music uploads. |
| Data lakes | Amazon S3 is an optimal foundation for a data lake because of its virtually unlimited scalability. |
| Swiggy - Capturing CDC using S3 data lake | Read more |
| ClickStream analytics using S3 on AWS | Read more |
| Feature | Remarks |
|---|---|
| ⭐ S3 vs Others | Read more |
| Buckets | Buckets are permanent containers that hold objects. |
| Storage Classes & Lifecycle | Amazon S3 offers a range of storage classes that you can choose from based on the data access, resiliency, and cost requirements of your workloads. |
| Highly durable | Amazon S3 is highly durable. |
| 🚀 Infinitely scalable | Application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in a bucket. Read more |
| Object Versioning | Versioning is a means of keeping multiple variants of an object in the same bucket. |
| Object Lock | With S3 Object Lock, you can store objects using a write-once-read-many (WORM) model. |
| Object Limit - Up to 5TB | When uploading data via the AWS Management Console, the maximum file that you can upload is 160GB. - To upload a file larger than 160 GB, use the AWS CLI, AWS SDK, or Amazon S3 REST API. Read more |
| S3 Notification | Amazon S3 can publish event notification messages to a destination (like Lambda, SQS, SNS etc.) on new object upload. |
| Multipart Upload | Multipart upload allows you to upload a single object as a set of parts. - Each part is a contiguous portion of the object's data. - You can upload these object parts independently and in any order. |
| Object Expiration | Object Expiration function allows you to define rules to schedule the removal of your objects after a pre-defined time period. - The rules are specified in the Lifecycle Configuration policy that you apply to a bucket. |
| 👍 Strong data consistency model | After a successful write of a new object or overwrite of an existing object, any subsequent read request immediately receives the latest version of the object. Read more |
| S3 Transfer Acceleration | S3 Transfer Acceleration can speed up content transfers to and from Amazon S3 by as much as 50-500% for long-distance transfer of larger objects. |
| Cross-Region Replication (CRR) | Cross-Region replication is a bucket-level configuration that enables automatic, asynchronous copying of objects across buckets in different AWS Regions. |
| Same-Region Replication (SRR) | Amazon S3 supports automatic and asynchronous replication of newly uploaded S3 objects to a destination bucket in the same AWS Region. Read more |
| Batch Replication | S3 Batch Replication provides you a way to replicate objects that existed before a replication configuration was in place, objects that have previously been replicated, and objects that have failed replication. |
| Object Metadata | Object metadata is set of name-value pairs with which you can store information regarding the object. |
| Byte Range Fetch | Using the Range HTTP header in a GET Object request, you can fetch a byte-range from an object, transferring only the specified portion. - A byte-range request is a perfect way to get the beginning of a file and ensuring we remain efficient during our scan of our S3 bucket. Read more |
| S3 Select | With Amazon S3 Select, you can use simple structured query language (SQL) statements to filter the contents of an Amazon S3 object and retrieve just the subset of data that you need. - By using Amazon S3 Select to filter this data, you can reduce the amount of data that Amazon S3 transfers, which reduces the cost and latency to retrieve this data. |
| Transfer Charges | There are no S3 data transfer charges when data is transferred in from the internet. |
- Instead of directly accessing the S3 resource, we should use CloudFront distribution in the middle (which acts as a CDN)
| Tool | Description |
|---|---|
| ⭐ Amazon S3 Storage Lens | Storage Lens helps in organization-wide visibility into Object Storage. |
| Analyse data using Amazon Athena | Amazon Athena is a serverless, interactive analytics service built on open-source frameworks, supporting open-table and file formats. |
| Analyse data using Amazon QuickSight | Amazon QuickSight powers data-driven organizations with unified business intelligence (BI) at hyperscale. |
| Amazon S3 analytics – Storage Class Analysis | By using Amazon S3 analytics, Storage Class Analysis you can analyze storage access patterns to help you decide when to transition the right data to the right storage class. - It helps to determine when to transition less frequently accessed STANDARD storage to the STANDARD_IA storage class. |
| Amazon S3 inventory | Amazon S3 inventory is one of the tools Amazon S3 provides to help manage your storage. You can use it to audit and report on the replication and encryption status of your objects for business, compliance, and regulatory needs. |
| Amazon S3 Server access logging | Amazon S3 Server access logging provides detailed records for the requests that are made to a bucket. |
- It's a best practice to use aws s3 commands for multipart uploads and downloads, because these aws s3 commands automatically perform multipart uploading and downloading based on the file size.
| Name | Command | Remarks |
|---|---|---|
| Copy file | aws s3 cp | In this command, you use the aws s3 cp command to copy the whale.jpg file from the client local machine in c:\sourcefiles to the demo-oceanlife bucket. |
| List files | aws s3 ls | List all files in S3 |
| List files in bucket | aws s3 ls bucketname | List all files in S3 bucket |
| Sync folder | aws s3 sync |
The aws S3 sync command uses the CopyObject APIs to copy objects between S3 buckets. - The sync command lists the source and target buckets to identify objects that are in the source bucket but that aren't in the target bucket. |
| Move file | aws s3 mv | Move S3 file from one bucket to another. |