reference s3 structure

참조 — S3 버킷 구조

s3://aegis-bucket-data의 현재 prefix 구조다. 실제 key builder와 Terraform 설정을 기준으로 정리한다.

---

버킷

항목	값
Bucket	aegis-bucket-data
Region	ap-south-1
Owner	infra/foundation
Public access	block public access
Object ownership	BucketOwnerEnforced
Default encryption	SSE-S3 AES256
Versioning	enabled

---

전체 Prefix

s3://aegis-bucket-data/
  raw/
  processed/
  processed_agg/
  image_snapshot/
  reports/daily/

---

raw/

IoT Rule이 MQTT payload를 SELECT *로 그대로 저장한다.

raw/{factory_id}/{source_type}/yyyy=YYYY/mm=MM/dd=DD/{message_id}.json

예시:

raw/factory-a/factory_state/yyyy=2026/mm=06/dd=08/factory-a:factory_state:worker2:2026-06-08T09:42:35Z.json
raw/factory-a/infra_state/yyyy=2026/mm=06/dd=08/factory-a:infra_state:cluster:2026-06-08T09:42:40Z.json
raw/factory-a/image_snapshot/yyyy=2026/mm=06/dd=08/factory-a:image_snapshot:worker2:2026-06-08T09:42:35Z.json

raw/는 source type별로 hh partition을 사용하지 않는다.

source_type	hh partition	body
factory_state	없음	canonical factory state JSON
infra_state	없음	canonical infra state JSON
image_snapshot	없음	image binary가 아니라 S3 image metadata JSON

---

processed/

DataProcessor가 정규화 결과와 상태 snapshot을 저장한다.

공통 key 형식:

processed/{factory_id}/{dataset}/yyyy=YYYY/mm=MM/dd=DD/hh=HH/{message_id_or_updated_at}.json

dataset:

Dataset	Key	생성 조건
factory_state	processed/{factory_id}/factory_state/yyyy=YYYY/mm=MM/dd=DD/hh=HH/{message_id}.json	factory_state 수신
risk_score	processed/{factory_id}/risk_score/yyyy=YYYY/mm=MM/dd=DD/hh=HH/{message_id}.json	factory_state 수신 후 risk 계산
infra_state	processed/{factory_id}/infra_state/yyyy=YYYY/mm=MM/dd=DD/hh=HH/{message_id}.json	infra_state 수신
image_snapshot	processed/{factory_id}/image_snapshot/yyyy=YYYY/mm=MM/dd=DD/hh=HH/{message_id}.json	image_snapshot metadata 수신
state_snapshot	processed/{factory_id}/state_snapshot/yyyy=YYYY/mm=MM/dd=DD/hh=HH/{updated_at}.json	factory_state, infra_state, refresh 처리

예시:

processed/factory-b/factory_state/yyyy=2026/mm=05/dd=21/hh=10/{message_id}.json
processed/factory-b/risk_score/yyyy=2026/mm=05/dd=21/hh=10/{message_id}.json
processed/factory-b/infra_state/yyyy=2026/mm=05/dd=21/hh=10/{message_id}.json
processed/factory-b/state_snapshot/yyyy=2026/mm=05/dd=21/hh=10/2026-05-21T10:00:03.123Z.json
processed/factory-a/image_snapshot/yyyy=2026/mm=06/dd=08/hh=09/factory-a:image_snapshot:worker2:2026-06-08T09:42:35Z.json

state_snapshot object는 DynamoDB HISTORY#STATE와 같은 상태 snapshot이지만, S3에는 DynamoDB TTL 필드인 ttl을 저장하지 않는다.

---

processed/cloud_infra/

CloudInfra collector가 cloud dashboard read model의 S3 장기 사본을 저장한다.

processed/cloud_infra/fast/yyyy=YYYY/mm=MM/dd=DD/hh=HH/{updated_at}.json
processed/cloud_infra/slow/yyyy=YYYY/mm=MM/dd=DD/hh=HH/{updated_at}.json

예시:

processed/cloud_infra/fast/yyyy=2026/mm=06/dd=01/hh=15/2026-06-01T15-30-00-000Z.json
processed/cloud_infra/slow/yyyy=2026/mm=06/dd=01/hh=15/2026-06-01T15-35-00-000Z.json

S3 object에는 DynamoDB TTL 필드인 ttl을 저장하지 않는다.

---

processed_agg/

GraphAggregator5m이 DynamoDB GRAPH#5M item의 장기 보조 object를 저장한다.

processed_agg/{factory_id}/metrics_5m/yyyy=YYYY/mm=MM/dd=DD/hh=HH/mm=MM.json

예시:

processed_agg/factory-b/metrics_5m/yyyy=2026/mm=05/dd=28/hh=10/mm=05.json

마지막 mm=MM.json의 MM은 5분 bucket 시작 minute이다.

---

image_snapshot/

AI event 이미지 원본 바이너리 저장 경로다. 이 prefix는 raw/processed metadata와 다르다.

image_snapshot/factory_id={factory_id}/yyyy=YYYY/mm=MM/dd=DD/hh=HH/{filename}

예시:

image_snapshot/factory_id=factory-a/yyyy=2026/mm=06/dd=08/hh=09/260608094235_event_FALLEN.jpg

생성 흐름:

snapshot-uploader
  -> SnapshotPresigner
  -> presigned S3 PUT
  -> image_snapshot/ binary
  -> outbox image_snapshot metadata
  -> raw/factory-a/image_snapshot/... metadata
  -> processed/factory-a/image_snapshot/... metadata
  -> DynamoDB LATEST.latest_image_snapshot

---

reports/daily/

Daily report pipeline 산출물이다.

reports/daily/yyyy=YYYY/mm=MM/dd=DD/{factory_id}/
reports/daily/yyyy=YYYY/mm=MM/dd=DD/cloud-infra/

factory report artifact:

reports/daily/yyyy=YYYY/mm=MM/dd=DD/{factory_id}/intermediate/hourly/hh=00.json
reports/daily/yyyy=YYYY/mm=MM/dd=DD/{factory_id}/factory-daily-summary.json
reports/daily/yyyy=YYYY/mm=MM/dd=DD/{factory_id}/report-context.json
reports/daily/yyyy=YYYY/mm=MM/dd=DD/{factory_id}/report.md
reports/daily/yyyy=YYYY/mm=MM/dd=DD/{factory_id}/generation-metadata.json

cloud infra report artifact:

reports/daily/yyyy=YYYY/mm=MM/dd=DD/cloud-infra/intermediate/hourly/hh=00.json
reports/daily/yyyy=YYYY/mm=MM/dd=DD/cloud-infra/cloud-infra-daily-summary.json
reports/daily/yyyy=YYYY/mm=MM/dd=DD/cloud-infra/report-context.json
reports/daily/yyyy=YYYY/mm=MM/dd=DD/cloud-infra/report.md
reports/daily/yyyy=YYYY/mm=MM/dd=DD/cloud-infra/generation-metadata.json

Dashboard Reports API는 reports/daily/yyyy=YYYY/mm=MM/dd=DD/{target}/report.md를 읽는다.

---

Partition 요약

Prefix	yyyy/mm/dd	hh	추가 partition
raw/{factory_id}/{source_type}/	있음	없음	없음
processed/{factory_id}/{dataset}/	있음	있음	없음
processed/cloud_infra/{fast,slow}/	있음	있음	없음
processed_agg/{factory_id}/metrics_5m/	있음	있음	mm=MM.json
image_snapshot/factory_id={factory_id}/	있음	있음	factory_id=...
reports/daily/	있음	hourly intermediate만 hh=HH.json	target {factory_id} 또는 cloud-infra

---

Lifecycle 요약

Prefix	정책
raw/	90일 후 Glacier Instant Retrieval 전환
processed/	365일 후 Standard-IA 전환
latest/ (legacy/non-contract)	과거 lifecycle 흔적. 현재 read model 계약 prefix가 아님
processed_agg/	별도 rule 없음
image_snapshot/	별도 rule 없음
reports/daily/	별도 rule 없음

---

확인 명령

aws s3 ls s3://aegis-bucket-data/raw/factory-a/ --recursive --region ap-south-1
aws s3 ls s3://aegis-bucket-data/processed/factory-a/ --recursive --region ap-south-1
aws s3 ls s3://aegis-bucket-data/processed_agg/factory-b/metrics_5m/ --recursive --region ap-south-1
aws s3 ls s3://aegis-bucket-data/image_snapshot/factory_id=factory-a/ --recursive --region ap-south-1
aws s3 ls s3://aegis-bucket-data/reports/daily/ --recursive --region ap-south-1

---

관련 문서

• 클라우드 스토리지
• 데이터 생명주기
• 데이터 read model