tuulos / disco

Disco 0.2.3 depends on erlang-base 13b, which Debian Lenny doesn't supply. The image "ami-e69d798f" is a lenny install, so it doesn't have the required packages.

Furthermore, setup-instances.py overrides /etc/apt/sources.list to only use discoproject and lenny repos, so it reinforces the same issue.

Running Disco 0.2.3 source distribution on ami-2946a740 (Debian Squeeze AMI from http://alestic.com/), with one master node and one remote slave, I was unable to run a job successfully. I went through all of the steps on the Troubleshooting Disco page, and found that the slave:start test failed with an {error, timeout} result. I tried a number of different flavors of node names (ec2 internal name, short names defined in /etc/hosts), and met with the same results. I was successfully able to run slave:start on the slave node to start a slave locally, but the same command from the master failed.

Adding new lines to the config table seems to drop old lines sometimes. Correspondingly, adding new nodes to the cluster on the fly doesn't always produce expected results.

You can easily make the worker run out of file descriptors e.g. by setting nr_reduces = 1000. This may result e.g. to spurious DNS failures, since no new sockets can be opened.

Disco should limit the number of fds used in a worker process, at least to make the error messages more descriptive.

If a job leaks memory / consumes vast amounts of memory otherwise, it can take nodes down easily. Master should monitor per-task/job/node memory consumption and kill rogue processes.

Raising the number of reduces to thousands is problematic because it requires a huge number of open file descriptors per process.

It should be possible to combine various low-level readers, such as gzip decoding, with higher level custom readers. This requires that you can specify a list (stack) of readers / writers instead of just one. See

http://groups.google.com/group/disco-dev/browse_thread/thread/7e63e6da57bcd05b?hl=en

Merge Eran Sandler's patches to add support for Gzipped inputs. See

http://groups.google.com/group/disco-dev/browse_thread/thread/7e63e6da57bcd05b?hl=en

When running tests map tasks execute ok even if lighttpd is down on a node. However, reduce fails since it can't access the map results. At least with comm_httplib this results to a cryptic error message - at least it could say nicely what's wrong.

Job fails fatally due to a glusterfs error (timeout), even though the error should be recoverable.

Gluster logs (client):

[2009-08-12 15:32:09] E [client-protocol.c:437:client_ping_timer_expired] dx26-vol1: Server 172.16.1.27:9900 has not responded in the last 60 seconds, disconnecting.

Server:
[2009-08-12 18:44:41] E [server-protocol.c:3903:server_readv] vol1: invalid argument: state->fd [2009-08-12 18:44:41] W [server-protocol.c:4210:server_fstat] server: fd - 1: unresolved fd [2009-08-12 18:44:41] E [server-protocol.c:5763:server_finodelk] server: fd - 1: unresolved fd

Disco:

Worker failed. Last words:
Traceback (most recent call last):
File "/usr/bin/disco-worker", line 69, in

method(m)

File "/var/lib/python-support/python2.5/disconode/disco_worker.py", line 392, in op_map

run_map(job_input[0], partitions, map_params)

File "/var/lib/python-support/python2.5/disconode/disco_worker.py", line 327, in run_map

for entry in reader:

File "/home/tuulos/src/disco/pydisco/disco/func.py", line 60, in netstr_reader File "/home/tuulos/src/disco/pydisco/disco/func.py", line 40, in read_netstr IOError: [Errno 107] Transport endpoint is not connected
close failed: [Errno 107] Transport endpoint is not connected

/usr/local/lib/erlang/erts-5.7.2/bin/epmd -daemon This one caused me a headache as I wasn't able to restart the master. The epmd process was still running and prevented master from starting up.

If Glusterfs fails on a single node, it is detected only when results are being moved to resultfs, which leads to a fatal error. However, the error should be recoverable by executing the task on another node.

http://groups.google.com/group/disco-dev/browse_thread/thread/8c0970c82587fa5d?hl=en

Integrate MochiWeb as a default web server to Disco, to make setting up the system easier. Lighttpd should be still provided as a high-performace option (Lighty could be supervised by Disco).

If a task fails due to e.g. a temporarily unavailable input, it should wait for some time before re-trying. Otherwise it can easily reach the maximum number of failed tasks in a few seconds.

Master should store job records persistently, so master could be restarted without losing any records.

The bin/disco script could include a troubleshoot mode which executes steps at

http://discoproject.org/doc/start/troubleshoot.html

automatically and shows suggestions how to fix detected issues.

disco-worker catches exceptions with the following code:

    try:
            run(method, mode, part, m)
    except comm.CommException, x:
            util.data_err("HTTP error: %s" % x)
    except IOError, x:
            util.data_err("IO error: %s" % x)

The calls to util.data_err fail because it requires 2 parameters.

If hundreds (>400) of jobs are running (and finishing) concurrently, gen_server:call(disco_server) calls can start to timeout.

On FreeBSD 7.2, the inet:gethostname() call returns the short hostname without the domain, e.g., "example" instead of "example.com." This causes the master node to fail with the error:

Error{EXIT,{{badmatch,{error,nxdomain}},[{handle_job,gethostname,0},{handle_job,set_disco_url,2},{handle_ctrl,handle,2},{scgi_server,handle_request,2},{scgi_server,scgi_worker,1}]}}

The solution is to patch handle_job.erl to account for incomplete hostnames and append the domain when it isn't present.

--- master/src/handle_job.erl.orig  2009-12-05 22:09:09.811589122 -0500
+++ master/src/handle_job.erl   2009-12-05 21:54:09.920556045 -0500
@@ -114,10 +114,14 @@
         end.
 
 gethostname() ->
-        {ok, Hostname} = inet:gethostname(),
-        {ok, Hostent}  = inet:gethostbyname(Hostname),
+        {ok, Host} = inet:gethostname(),
+        case inet_db:res_option(domain) of
+             "" -> Hostname = Host; 
+             Domain -> Hostname = Host ++ "." ++ Domain
+        end,
+        {ok, Hostent}  = inet:gethostbyname(Hostname),
         Hostent#hostent.h_name.
-        
+
 set_disco_url(undefined, Msg) ->
         {value, {_, SPort}} =
                 lists:keysearch(<<"SERVER_PORT">>, 1, Msg),

This code was taken from http://web.mit.edu/erlang_vR11B-5/arch/i386_linux26/lib/erlang/lib/kernel-2.11.5/src/net_adm.erl

Basically, one part passes a unicode string to setopt in pycurl. I recommend removing pycurl as dependency for disco. Traceback is below:

URL: u'http://localhost:8989/localhost/77/disco_tut@1260921574/reduce-index.txt'
Traceback (most recent call last):
  File "count_words.py", line 24, in <module>
    for word, total in result_iterator(results):
  File "...disco-0.2.3/pydisco/disco/core.py", line 466, in result_iterator
    sze, fd = comm.open_remote(url)
  File ".../disco-0.2.3/pydisco/disco/comm_curl.py", line 149, in open_remote
    conn = CurlConn(url, handle = c, expect = expect)
  File ".../disco-0.2.3/pydisco/disco/comm_curl.py", line 66, in __init__
    self.init_handle(url)
  File ".../disco-0.2.3/pydisco/disco/comm_curl.py", line 89, in init_handle
    self.handle.setopt(URL, url)
TypeError: invalid arguments to setopt

In OOBTestCase specifically, input_2 calls job_1...DiscoMultiJobTestFixture tests hasattr(..., 'input_2') which fails if job_1 fails, causing uninformative error.

All properties in Disco should be side-effect free and DiscoMultiJobTestFixture should be reworked to make it less complicated

ddfs should be distributed as part of disco and used internally

redesign the web API, including new things like (filtered) events

build some monitoring graphs using events API

allow users to not wait for all maps to finish before reduce/results phase

maps should be able to open/write to partitions dynamically (s.t. the number does not need to be predetermined)

original discodex implemented metadata server, take another look at what can be replaced by ddfs

overall, error visibility from web ui should be improved (think django-like debug pages)

use disco to execute arbitrary functions on all of its worker nodes