Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

S3 support #333

Closed
wants to merge 160 commits into
from

Conversation

Projects
None yet
4 participants

No description provided.

srobertson and others added some commits Jul 3, 2011

@srobertson srobertson Moved setup.py to the root directory to make pip happy. 373bbd8
@srobertson srobertson Code that doesn't have a module is obviously not part of the stdlib 9eec356
@srobertson srobertson Merge remote-tracking branch 'upstream/master' cfb5a2a
@srobertson srobertson upstream merge f4eb768
root make ddfs.pull respect the proxy fe5ac05
root fix hanging on chunked encoding be779cb
@srobertson srobertson Adding https support 537e28e
@srobertson srobertson https support for pycurl 9244ff5
root fixed typos, damn me for using edit in github 85f72cf
@srobertson srobertson Modified makefile to use setup.py in the root of the project 38f760e
@srobertson srobertson fixed tab in makefile 9895824
@pmundkur @srobertson pmundkur + srobertson Add some type definitions. 2b9a4ce
@pmundkur @srobertson pmundkur + srobertson Add specs for exported functions. dc28463
@pmundkur @srobertson pmundkur + srobertson Add specs for exported functions. 614bec2
@pmundkur @srobertson pmundkur + srobertson Add specs for exported functions. 13f5906
@pmundkur @srobertson pmundkur + srobertson Add specs for exported functions. 99952cb
@pmundkur @srobertson pmundkur + srobertson Code cleanup based on earlier tidier idioms. 6b76bcd
@pmundkur @srobertson pmundkur + srobertson Add specs for exported functions. 089afdf
@pmundkur @srobertson pmundkur + srobertson Add util types for spec-cing gen_server callbacks. bf126d2
@pmundkur @srobertson pmundkur + srobertson Use the util types in gen_server callback specs. 6998ed1
@pmundkur @srobertson pmundkur + srobertson Move some utility types to gs_utils to be more useful. 24d103b
@pmundkur @srobertson pmundkur + srobertson Cleanup of lock_server: api, specs for exported functions, tidier. 2efab79
@pmundkur @srobertson pmundkur + srobertson Remove some unused stop() functions. a3baac1
@pmundkur @srobertson pmundkur + srobertson Add specs for exported functions. 711ca00
@pmundkur @srobertson pmundkur + srobertson Add specs for exported functions. e43286b
@pmundkur @srobertson pmundkur + srobertson Add specs for exported functions. 24c00cb
@pmundkur @srobertson pmundkur + srobertson Add specs for exported functions. 0b110b6
@pmundkur @srobertson pmundkur + srobertson Add specs for exported functions. 3f512f2
@pmundkur @srobertson pmundkur + srobertson Add specs for exported functions. 551d859
@pmundkur @srobertson pmundkur + srobertson Remove unused function. 29df654
@pmundkur @srobertson pmundkur + srobertson Add specs for exported functions. 6e9efee
@pmundkur @srobertson pmundkur + srobertson Minor type tweaks. 8959606
@pmundkur @srobertson pmundkur + srobertson Add some exported types. 1bef84b
@pmundkur @srobertson pmundkur + srobertson Tidy code a bit. a2344a1
@pmundkur @srobertson pmundkur + srobertson Tidy code a bit. eafb551
@pmundkur @srobertson pmundkur + srobertson Tidy code a bit. 690c10d
@pmundkur @srobertson pmundkur + srobertson Tidy code a bit. ec28025
@pmundkur @srobertson pmundkur + srobertson Minor cleanup: use macro for module name. 7c69510
@pmundkur @srobertson pmundkur + srobertson Tidy code a bit. 9fcbeb1
@pmundkur @srobertson pmundkur + srobertson Tidy code in the scheduler policies. 62b2186
@pmundkur @srobertson pmundkur + srobertson Clarify some types. 88f07ae
@pmundkur @srobertson pmundkur + srobertson Add specs for exported functions. 1bc4bef
@pmundkur @srobertson pmundkur + srobertson Add specs for exported functions. aec6290
@pmundkur @srobertson pmundkur + srobertson More spec cleanups. c4c34bb
@pmundkur @srobertson pmundkur + srobertson Fix some specs. cbd666a
@pmundkur @srobertson pmundkur + srobertson Add specs for exported functions in event_server, and fix uncovered i…
…ssues.
a04e6a6
@pmundkur @srobertson pmundkur + srobertson Update customized zip module to newer version. 67cbf4c
@pmundkur @srobertson pmundkur + srobertson Enable warn_missing_spec warning. 465029e
@pmundkur @srobertson pmundkur + srobertson Handle invalid paths in ddfs_get. 64cf3d6
@pmundkur @srobertson pmundkur + srobertson Create DDFS root dir if needed. 749ae69
@pmundkur @srobertson pmundkur + srobertson Fix ddfs cat/xcat in client proxy mode. d0309bc
@pmundkur @srobertson pmundkur + srobertson Clarify proxy settings docs. 112d117
@pmundkur @srobertson pmundkur + srobertson Add proxy config to admin docs. 318141d
@pmundkur @srobertson pmundkur + srobertson Fix some case-catch instances. cd82ebe
@pmundkur @srobertson pmundkur + srobertson Avoid using client-side settings in master. 5e5b69c
@pmundkur @srobertson pmundkur + srobertson Fix a spec. b1e20e5
@pmundkur @srobertson pmundkur + srobertson Move some node start settings into disco_server. bebad0e
@pmundkur @srobertson pmundkur + srobertson Remove unused code. 67773fe
@pmundkur @srobertson pmundkur + srobertson Improve temp_gc interface. a37c6ed
@pmundkur @srobertson pmundkur + srobertson Improve web module interface for disco and ddfs. 1191b03

pmundkur and others added some commits Jun 6, 2012

@pmundkur @srobertson pmundkur + srobertson Remove another case-catch. 87459b3
@pmundkur @srobertson pmundkur + srobertson Remove another case-catch. 743c2b9
@pmundkur @srobertson pmundkur + srobertson Remove some more case-catch uses. bdd0f26
@pmundkur @srobertson pmundkur + srobertson Remove another case-catch. a8568c6
@pmundkur @srobertson pmundkur + srobertson Fix a spec. b7e1153
@pmundkur @srobertson pmundkur + srobertson Remove another case-catch. 4f81846
@pmundkur @srobertson pmundkur + srobertson Fix a spec. ffdc150
@pmundkur @srobertson pmundkur + srobertson Remove another case-catch. 1b47952
@pmundkur @srobertson pmundkur + srobertson Remove another case-catch. 1a83b8a
@pmundkur @srobertson pmundkur + srobertson Remove another case-catch, and clean up some specs. 757c64c
@pmundkur @srobertson pmundkur + srobertson Cleanup some specs. 45494bd
@pmundkur @srobertson pmundkur + srobertson Remove another case-catch. a25c47f
@pmundkur @srobertson pmundkur + srobertson Remove another case-catch. 2fa7683
@pmundkur @srobertson pmundkur + srobertson Remove another case-catch. f89d45c
@pmundkur @srobertson pmundkur + srobertson Remove another case-catch. 904a1a0
@pmundkur @srobertson pmundkur + srobertson Remove another case-catch. abb8444
@pmundkur @srobertson pmundkur + srobertson Remove another case-catch. 438aa81
@pmundkur @srobertson pmundkur + srobertson Move url() to common_types. a0adec8
@pmundkur @srobertson pmundkur + srobertson Move job_coordinator:preferred_host to disco:preferred_host. ea06561
@pmundkur @srobertson pmundkur + srobertson Define utility type. 853b408
@pmundkur @srobertson pmundkur + srobertson Rearrange some code in jobpack. 6126764
@pmundkur @srobertson pmundkur + srobertson Ensure jobpack:valid catches all errors, so that subsequent metadata …
…ops cannot error out.
b5c7a64
@pmundkur @srobertson pmundkur + srobertson Properly handle thrown errors during job initialization. 62a41d8
@pmundkur @srobertson pmundkur + srobertson Correct a spec and export a useful type. 35e11a9
@pmundkur @srobertson pmundkur + srobertson Move choose_random to disco_util from ddfs_util. 20e9ce3
@pmundkur @srobertson pmundkur + srobertson Change an error-exception throw into an error code. 2fb1a99
@pmundkur @srobertson pmundkur + srobertson Clarify return value of preferred_host and its return type for common…
… use.
850ff82
@pmundkur @srobertson pmundkur + srobertson Strengthen the groupby spec. 29605a9
@pmundkur @srobertson pmundkur + srobertson Use a proper api for the fair scheduler and remove direct gen_server …
…calls; fix some spec errors.
7bb97c6
@pmundkur @srobertson pmundkur + srobertson Use a proper api for the fair scheduler job. 7743522
@pmundkur @srobertson pmundkur + srobertson Some whitespace cleanup for clarity (almost no semantic changes, exce…
…pt for a minor init fix).
530736d
@pmundkur @srobertson pmundkur + srobertson Log replication req # to track replication progress. d90e299
@pmundkur @srobertson pmundkur + srobertson Log an unusual exception in ddfs_get; it can catch misconfigured prox…
…y setting.
3b7d3b5
@pmundkur @srobertson pmundkur + srobertson Fix result_iterator properly to handle proxied inputs. 712a07c
@pmundkur @srobertson pmundkur + srobertson Avoid unicode issues in partition function. Thanks to quasiben and tm…
…ielika.
55df9d2
@tsloughter @srobertson tsloughter + srobertson uploads to s3 for s3 urls returns by master 1ea4c1a
@tsloughter @srobertson tsloughter + srobertson s3 support configured through env variablesDISCO_USE_S3 and DISCO_S3_…
…BUCKET
12c9257
@tsloughter @srobertson tsloughter + srobertson backup tag data to s3 4acc36d
@tsloughter @srobertson tsloughter + srobertson writes tags to s3 d2c57b8
@tsloughter @srobertson tsloughter + srobertson reads tags from s3 if missing 716f9c4
@tsloughter @srobertson tsloughter + srobertson added scheme_s3 so on s3 url python worker can access data 3a1ec93
@pmundkur @srobertson pmundkur + srobertson Add some logging to track replication progress. 5ef9da3
@pmundkur @srobertson pmundkur + srobertson Change partition to use a pure hash, instead of converting to string/…
…unicode.
7c4c44a
@pmundkur @srobertson pmundkur + srobertson Fix a format issue when printing some worker errors. 7591e25
@pmundkur @srobertson pmundkur + srobertson Fix a type in the scheduler. e0d4d8e
@pmundkur @srobertson pmundkur + srobertson Fix a argument bug introduced by a case-catch fix. b0da35c
@pmundkur @srobertson pmundkur + srobertson Fix a typo in the docs. 3029446
@tsloughter @srobertson tsloughter + srobertson gc for s3 works cf36eb2
Owner

pmundkur commented Oct 8, 2012

Very cool work, thanks! Would it be possible to rebase this patchset on current master and get a cleaner patch series? This is going to be hard to pull: e.g. ef7c5a0 has new code along with upstream merge.

Crazy, did not mean to submit that yet. Guess we're horrible at using git.

Feel free to delete this pull request as the s3 stuff is very alpha. We'll get you a cleaner one in the near future.

Since the cat's out of the bag, have a look at http://triv.io. We're setting up a hosted version of disco, with a slew of goodies, most of which we'll be contributing back as soon as we're not embarrassed by them.

Owner

pmundkur commented Oct 12, 2012

Great, looking forward to it! And good luck!

Owner

pooya commented Jun 6, 2014

This PR cannot be merged in its current form.

@pooya pooya closed this Jun 6, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment