Skip to content

Latest commit

 

History

History
112 lines (80 loc) · 2.22 KB

README.md

File metadata and controls

112 lines (80 loc) · 2.22 KB
Raw

Youtube Comments Crawler

This function is youtube comment crawler to use AWS SAM.

Description

This function has the following configuration.

AWS Diagram

This function is use under AWS Service

  • Compute
    • Lambda
      • Function Exec
  • Database
    • DynamoDB
      • Resource Storage
  • Administration & Security
    • CloudWatch
      • Schedule Event, and Logging
  • Deployment & Management
    • CloudFormation
      • Create deploy stack
    • X-Ray
      • trace func exec and aws services

Requirement

  • aws-sam-cli
    • 0.14.2
  • go
    • 1.12.1
  • docker
    • 18.09.2
  • docker-compose
    • 1.23.2
  • aws-cli
    • 1.16.130

Usage

Development

Development use aws-sam-cli. So, Please install aws-sam-cli.

  • use Mac or Linux
$ brew tap aws/tap
$ brew install aws-sam-cli
  • using pip
$ pip install --user aws-sam-cli

Before execute commands, you need to get google service account with YouTube Data API certification enabled. After clone this repository, the following command will be executed.

  • container up
$ cd /path/to/youtube-comments-crawler
$ make create-network && docker-compose up -d
  • put item
$  make put-item TABLE_NAME='YoutubeCommentsCrawlerYoutubers' ITEM='{ "id": { "S": "unique xid insert" }, "name": { "S": "Please Input Youtuber Name" }, "channel_id": { "S": "Please Input Youtuber Channel ID" }}'
  • create dummy tables
 $ make create-table TABLE_NAME="YoutubeCommentsCrawlerVideos"
 $ make create-table TABLE_NAME="YoutubeCommentsCrawlerComments"
 $ make create-table TABLE_NAME="YoutubeCommentsCrawlerYoutubers"
  • execute lambda local
$ cd /path/to/youtube-comments-crawler
$ aws s3 cp /path/to/service-account.json s3://google-service-accounts-dev/youtube-comments-crawler \
--endpoint-url=http://localhost:9001 \
--region ap-northeast-1 --profile $MINIO_PROFILE

Deploy

Deploy is direct create croudformation stack. Inside calls sam package, sam deploy. Please Look at Makefile.

$ make create-package
$ make deploy-package

Install

git clone https://github.com/gtongy/youtube-comments-crawler

Author

gtongy