Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

S3 support #333

Closed
wants to merge 160 commits into from

4 participants

@tsloughter

No description provided.

srobertson and others added some commits
@srobertson srobertson Moved setup.py to the root directory to make pip happy.
373bbd8
@srobertson srobertson Code that doesn't have a module is obviously not part of the stdlib
9eec356
@srobertson srobertson Merge remote-tracking branch 'upstream/master'
cfb5a2a
@srobertson srobertson upstream merge
f4eb768
root make ddfs.pull respect the proxy
fe5ac05
root fix hanging on chunked encoding
be779cb
@srobertson srobertson Adding https support
537e28e
@srobertson srobertson https support for pycurl
9244ff5
root fixed typos, damn me for using edit in github
85f72cf
@srobertson srobertson Modified makefile to use setup.py in the root of the project
38f760e
@srobertson srobertson fixed tab in makefile
9895824
@pmundkur pmundkur Add some type definitions.
2b9a4ce
@pmundkur pmundkur Add specs for exported functions.
dc28463
@pmundkur pmundkur Add specs for exported functions.
614bec2
@pmundkur pmundkur Add specs for exported functions.
13f5906
@pmundkur pmundkur Add specs for exported functions.
99952cb
@pmundkur pmundkur Code cleanup based on earlier tidier idioms.
6b76bcd
@pmundkur pmundkur Add specs for exported functions.
089afdf
@pmundkur pmundkur Add util types for spec-cing gen_server callbacks.
bf126d2
@pmundkur pmundkur Use the util types in gen_server callback specs.
6998ed1
@pmundkur pmundkur Move some utility types to gs_utils to be more useful.
24d103b
@pmundkur pmundkur Cleanup of lock_server: api, specs for exported functions, tidier.
2efab79
@pmundkur pmundkur Remove some unused stop() functions.
a3baac1
@pmundkur pmundkur Add specs for exported functions.
711ca00
@pmundkur pmundkur Add specs for exported functions.
e43286b
@pmundkur pmundkur Add specs for exported functions.
24c00cb
@pmundkur pmundkur Add specs for exported functions.
0b110b6
@pmundkur pmundkur Add specs for exported functions.
3f512f2
@pmundkur pmundkur Add specs for exported functions.
551d859
@pmundkur pmundkur Remove unused function.
29df654
@pmundkur pmundkur Add specs for exported functions.
6e9efee
@pmundkur pmundkur Minor type tweaks.
8959606
@pmundkur pmundkur Add some exported types.
1bef84b
@pmundkur pmundkur Tidy code a bit.
a2344a1
@pmundkur pmundkur Tidy code a bit.
eafb551
@pmundkur pmundkur Tidy code a bit.
690c10d
@pmundkur pmundkur Tidy code a bit.
ec28025
@pmundkur pmundkur Minor cleanup: use macro for module name.
7c69510
@pmundkur pmundkur Tidy code a bit.
9fcbeb1
@pmundkur pmundkur Tidy code in the scheduler policies.
62b2186
@pmundkur pmundkur Clarify some types.
88f07ae
@pmundkur pmundkur Add specs for exported functions.
1bc4bef
@pmundkur pmundkur Add specs for exported functions.
aec6290
@pmundkur pmundkur More spec cleanups.
c4c34bb
@pmundkur pmundkur Fix some specs.
cbd666a
@pmundkur pmundkur Add specs for exported functions in event_server, and fix uncovered i…
…ssues.
a04e6a6
@pmundkur pmundkur Update customized zip module to newer version.
67cbf4c
@pmundkur pmundkur Enable warn_missing_spec warning.
465029e
@pmundkur pmundkur Fix ddfs cat/xcat in client proxy mode.
d0309bc
@pmundkur pmundkur Handle invalid paths in ddfs_get.
64cf3d6
@pmundkur pmundkur Create DDFS root dir if needed.
749ae69
@pmundkur pmundkur Clarify proxy settings docs.
112d117
@pmundkur pmundkur Add proxy config to admin docs.
318141d
@pmundkur pmundkur Fix some case-catch instances.
cd82ebe
@pmundkur pmundkur Avoid using client-side settings in master.
5e5b69c
@pmundkur pmundkur Fix a spec.
b1e20e5
@pmundkur pmundkur Move some node start settings into disco_server.
bebad0e
@pmundkur pmundkur Remove unused code.
67773fe
@pmundkur pmundkur Improve temp_gc interface.
a37c6ed
@pmundkur pmundkur Improve web module interface for disco and ddfs.
1191b03
@pmundkur pmundkur Implement port-mapping for cluster-in-a-box mode, based on work by Ha…
…rry Nakos.
28396e2
@pmundkur pmundkur Implement node naming for local-cluster mode (cluster-in-a-box), base…
…d on work by Harry Nakos.
85524be
@pmundkur pmundkur Update proxy rules for local-cluster mode.
0df108c
@pmundkur pmundkur More meaningful ddfs node/get/put startup messages.
fbf8b4d
@pmundkur pmundkur Track how long a node has been up.
a4e8254
@pmundkur pmundkur Require an explicit ack from node before marking it up.
2c720ec
@pmundkur pmundkur Tidy up and rationalize some more specs.
f66728a
@pmundkur pmundkur Use an atom instead of string for task mode.
a1ca1b5
@pmundkur pmundkur Reenable warnings_as_errors.
e90662b
@pmundkur pmundkur Correctly handle worker node on master in proxy mode.
8b9d927
@pmundkur pmundkur Cleanup event_server api to avoid using direct gen_server calls.
dd1f2f5
@pmundkur pmundkur Kill proxy on clean termination.
07025da
@pmundkur pmundkur Tidy up event_server.
b5d1f30
@pmundkur pmundkur Fix a spec.
c652006
@pmundkur pmundkur Automatically start proxy in local_cluster mode.
130ac11
@pmundkur pmundkur Minor spec cleanup in ddfs_master.
780af61
@pmundkur pmundkur Minor spec cleanup in ddfs.
3427a71
@pmundkur pmundkur Some cleanup of internal api between ddfs and ddfs_master.
1d65910
@pmundkur pmundkur Log ddfs get/put ports on node startup.
2580b02
@pmundkur pmundkur Properly update proxy on config changes in local cluster mode.
7f07656
@pmundkur pmundkur Fix replication in local cluster mode.
8233845
@pmundkur pmundkur Minor renaming.
cf55fec
@pmundkur pmundkur Fixup event_server / ui task stat calculations.
5102b8c
@pmundkur pmundkur Use lager:safe_format for disco:format.
6120187
@pmundkur pmundkur Omit now-redundant format safety check.
db69acf
@pmundkur pmundkur Remove unused module.
9bec876
@pmundkur pmundkur Rearrange event server code for clarity.
3c15c94
@pmundkur pmundkur Move out event-server logic into seperate handlers.
7382b63
@pmundkur pmundkur Minor event-server api renaming for clarity.
1f6278c
@pmundkur pmundkur Strengthen event spec in event_server api.
3bbfa24
@pmundkur pmundkur Tidy up internal apis in job_coordinator.
b72e76d
@pmundkur pmundkur More minor event-server cleanup.
87092ac
@pmundkur pmundkur Remove another unused module.
fd089ba
@pmundkur pmundkur Modify event_server state to store task stats directly.
a0f611e
@pmundkur pmundkur Use defaults when returning jobinfo for jobs still initializing (#304).
6d8ca81
@pmundkur pmundkur Remove another case-catch.
03ae2ba
@pmundkur pmundkur Remove another case-catch.
e8b6b8f
@pmundkur pmundkur Remove another case-catch.
ce8893d
@pmundkur pmundkur Remove another case-catch.
4cd1206
@pmundkur pmundkur Remove another case-catch.
1c59de0
@pmundkur pmundkur Remove another case-catch.
3ee6918
@pmundkur pmundkur Remove another case-catch.
cc993b9
@pmundkur pmundkur Remove another case-catch.
5cce194
@pmundkur pmundkur Fix a spec.
231b1dc
@pmundkur pmundkur Remove another case-catch.
9489301
@pmundkur pmundkur Remove another case-catch.
87459b3
@pmundkur pmundkur Remove another case-catch.
743c2b9
@pmundkur pmundkur Remove some more case-catch uses.
bdd0f26
@pmundkur pmundkur Remove another case-catch.
a8568c6
@pmundkur pmundkur Fix a spec.
b7e1153
@pmundkur pmundkur Remove another case-catch.
4f81846
@pmundkur pmundkur Fix a spec.
ffdc150
@pmundkur pmundkur Remove another case-catch.
1b47952
@pmundkur pmundkur Remove another case-catch.
1a83b8a
@pmundkur pmundkur Remove another case-catch, and clean up some specs.
757c64c
@pmundkur pmundkur Cleanup some specs.
45494bd
@pmundkur pmundkur Remove another case-catch.
a25c47f
@pmundkur pmundkur Remove another case-catch.
2fa7683
@pmundkur pmundkur Remove another case-catch.
f89d45c
@pmundkur pmundkur Remove another case-catch.
904a1a0
@pmundkur pmundkur Remove another case-catch.
abb8444
@pmundkur pmundkur Remove another case-catch.
438aa81
@pmundkur pmundkur Move url() to common_types.
a0adec8
@pmundkur pmundkur Move job_coordinator:preferred_host to disco:preferred_host.
ea06561
@pmundkur pmundkur Define utility type.
853b408
@pmundkur pmundkur Rearrange some code in jobpack.
6126764
@pmundkur pmundkur Ensure jobpack:valid catches all errors, so that subsequent metadata …
…ops cannot error out.
b5c7a64
@pmundkur pmundkur Properly handle thrown errors during job initialization.
62a41d8
@pmundkur pmundkur Correct a spec and export a useful type.
35e11a9
@pmundkur pmundkur Move choose_random to disco_util from ddfs_util.
20e9ce3
@pmundkur pmundkur Change an error-exception throw into an error code.
2fb1a99
@pmundkur pmundkur Clarify return value of preferred_host and its return type for common…
… use.
850ff82
@pmundkur pmundkur Strengthen the groupby spec.
29605a9
@pmundkur pmundkur Use a proper api for the fair scheduler and remove direct gen_server …
…calls; fix some spec errors.
7bb97c6
@pmundkur pmundkur Use a proper api for the fair scheduler job.
7743522
@pmundkur pmundkur Some whitespace cleanup for clarity (almost no semantic changes, exce…
…pt for a minor init fix).
530736d
@pmundkur pmundkur Log an unusual exception in ddfs_get; it can catch misconfigured prox…
…y setting.
3b7d3b5
@pmundkur pmundkur Fix result_iterator properly to handle proxied inputs.
712a07c
@pmundkur pmundkur Avoid unicode issues in partition function. Thanks to quasiben and tm…
…ielika.
55df9d2
@pmundkur pmundkur Log replication req # to track replication progress.
d90e299
@tsloughter tsloughter uploads to s3 for s3 urls returns by master
1ea4c1a
@tsloughter tsloughter s3 support configured through env variablesDISCO_USE_S3 and DISCO_S3_…
…BUCKET
12c9257
@tsloughter tsloughter backup tag data to s3
4acc36d
@tsloughter tsloughter writes tags to s3
d2c57b8
@tsloughter tsloughter reads tags from s3 if missing
716f9c4
@tsloughter tsloughter added scheme_s3 so on s3 url python worker can access data
3a1ec93
@pmundkur pmundkur Add some logging to track replication progress.
5ef9da3
@pmundkur pmundkur Change partition to use a pure hash, instead of converting to string/…
…unicode.
7c4c44a
@pmundkur pmundkur Fix a format issue when printing some worker errors.
7591e25
@pmundkur pmundkur Fix a type in the scheduler.
e0d4d8e
@pmundkur pmundkur Fix a argument bug introduced by a case-catch fix.
b0da35c
@pmundkur pmundkur Fix a typo in the docs.
3029446
@tsloughter tsloughter gc for s3 works
cf36eb2
@pmundkur
Owner

Very cool work, thanks! Would it be possible to rebase this patchset on current master and get a cleaner patch series? This is going to be hard to pull: e.g. ef7c5a0 has new code along with upstream merge.

@srobertson

Crazy, did not mean to submit that yet. Guess we're horrible at using git.

Feel free to delete this pull request as the s3 stuff is very alpha. We'll get you a cleaner one in the near future.

Since the cat's out of the bag, have a look at http://triv.io. We're setting up a hosted version of disco, with a slew of goodies, most of which we'll be contributing back as soon as we're not embarrassed by them.

@pmundkur
Owner

Great, looking forward to it! And good luck!

@pooya
Owner

This PR cannot be merged in its current form.

@pooya pooya closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.