Permalink
Browse files

write data to new GS (data.gharchive.org) bucket

  • Loading branch information...
igrigorik committed Apr 3, 2018
1 parent 195c2a2 commit 7ada8644d842e97f5bf90289edf47546d9dc03f4
Showing with 1 addition and 1 deletion.
  1. +1 −1 crawler/tasks.cron
View
@@ -1,5 +1,5 @@
# sync data to cloud storage
-5 * * * * /bin/bash -l -c 'cd /home/igrigorik/githubarchive.org/crawler/data && gzip -9 `date +"\%Y-\%m-\%d-\%-k" -d "1 hour ago"`.json && gsutil -h "Content-Type: application/x-gzip" cp -a public-read `date +"\%Y-\%m-\%d-\%-k" -d "1 hour ago"`.json.gz gs://data.githubarchive.org/' >> /home/igrigorik/githubarchive.org/crawler/data/sync.log 2>&1
+5 * * * * /bin/bash -l -c 'cd /home/igrigorik/githubarchive.org/crawler/data && gzip -9 `date +"\%Y-\%m-\%d-\%-k" -d "1 hour ago"`.json && gsutil -h "Content-Type: application/x-gzip" cp -a public-read `date +"\%Y-\%m-\%d-\%-k" -d "1 hour ago"`.json.gz gs://data.gharchive.org/' >> /home/igrigorik/githubarchive.org/crawler/data/sync.log 2>&1
# import data into bigquery
8 * * * * /bin/bash -l -c 'cd /home/igrigorik/githubarchive.org/bigquery && ruby upload.rb -i ../crawler/data/`date +"\%Y-\%m-\%d-\%-k" -d "1 hour ago"`.json.gz' >> /home/igrigorik/githubarchive.org/crawler/data/sync.log 2>&1

0 comments on commit 7ada864

Please sign in to comment.