Skip to content

Commit

Permalink
cp large mapside in-memory lookup file from s3
Browse files Browse the repository at this point in the history
  • Loading branch information
Pete Skomoroch committed Mar 29, 2010
1 parent 59b1f3a commit 21a4315
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion pig/spatialtrends/tweet_ngrams.pig
Expand Up @@ -66,8 +66,11 @@ LOWER($0) as tweet_text;
-- 31055 5074472 Wed Feb 10 04:59:42 +0000 2010 thanks for coming to pub quiz steph jess ali and stacey!
-- 06073 5391811 Wed Feb 10 04:50:26 +0000 2010 looooooooost!!

-- fetch larger file from S3 for shipping to distributed cache
cp s3://where20demo/wikiphrases.pkl file:///mnt/

DEFINE tweet_tokenizer `tweet_tokenizer.py`
SHIP ('tweet_tokenizer.py', 'nltkandyaml.mod', 's3://where20demo/wikiphrases.pkl');
SHIP ('tweet_tokenizer.py', 'nltkandyaml.mod', '/mnt/wikiphrases.pkl');
tweet_ngrams = STREAM std_location_tweets THROUGH tweet_tokenizer
AS (ngram:chararray, fipscode:chararray, geonameid:int, date:chararray, hour:int, daily_trend:float);

Expand Down

0 comments on commit 21a4315

Please sign in to comment.