Permalink
Commits on Jan 4, 2011
  1. Release neubot/0.3.2

    bassosimone committed Jan 4, 2011
Commits on Jan 3, 2011
  1. neubot/speedtest.py: Periodically log the queue length

    bassosimone committed Dec 13, 2010
    This commit is very useful to monitor the amount of clients waiting
    in queue for their turn to perform the speedtest.  If the queue lenght
    becomes -on the average- too big we need either to add more servers
    or to tweak (again) the scheduling algorithm.
  2. neubot/speedtest.py: Make sure first session's connection is not special

    bassosimone committed Nov 26, 2010
    The algorithm that maps a connection to the connection's session unique
    identifier runs when we have read just the headers.  Therefore the first
    connection of a session could not be mapped because at that point the
    session does not exist.  But we want to track that connection too.  In
    order to do that, this patch adds a special case that register a new
    connection if this is the first negotiation of the related session.
  3. neubot/speedtest.py: Use connection_lost to improve queue algorithm

    bassosimone committed Nov 26, 2010
    Add _SessionTrackerMixin a dictionary that maps each connection to
    the identifier that identifies it.  Refresh the binding each time
    we receive the headers of the new request (*please note that the
    first connection would be skipped using the algorithm of this
    patch*).  And, when a connection is lost, try to use the identifier
    to access the related session and, if the session exist then remove
    it (the first connection that terminates remove the session and
    all the others will find that the session does not exist.)
  4. neubot/speedtest.py: Improve queue algorithm

    bassosimone committed Nov 24, 2010
    The main problem with the existing queue algorithm is that it is
    not scalable enough, because it allows just one transmission test
    at once.  This patch tries to improve the queue algorithm to support
    up to three concurrent tests (actually three is just a parameter
    that could be raised at will.)
    
    The existing algorithm
    ``````````````````````
    
    The existing algorithm keeps track of the clients using a (fifo)
    queue that contains the unique identifier of each client.  This
    identifier is an uuid4 chosen by the server and assigned to the
    client.  The client MUST pass its unique identifier to the server
    using the 'authorization' header.
    
    In check_request_headers() the server checks whether the URI that
    the client wants to access is restricted or not.  Basically, all
    URIs are restricted but the one used to negotiate.  If the URI is
    restricted and there is not the unique identifier or the unique
    identifier passed by the client is not the first in the queue, the
    connection is closed.  So, there is the implied assumption that
    just the first client in queue is authorized to take the test.
    
    In do_negotiate() the server will generate an unique identifier for
    the client, if the client did not provided one.  Then the client
    will be granted the authorization if and only if it is the first
    one in queue (as said above, the first one is the authorized one.)
    
    The function to calculate the position in queue of each client is
    very bad, because it will scan the list for each client--and so its
    complexity is roughly O(N^2).
    
    When a client uploads the results to /speedtest/collect, the server
    invokes _speedtest_complete().  This function will pop the first
    element from the queue and start a new negotiation.
    
    To avoid starvation, the function _speedtest_check_timeout() is
    invoked every 3 seconds.  If it finds that the first client in queue
    has not completed the test in 30 seconds, this function will pop
    the first element from the queue and start a new negotiation.
    
    This is bad because if the connection with the first client in
    queue is lost, the queue will be blocked for 30 seconds (in the
    worst case) because the `connection lost` event is not used to
    trigger a new negotiation.
    
    The new algorithm
    `````````````````
    
    There is a class, SessionState, for each connected client, and
    this class contains the following fields:
    
    active
      whether this client is performing a test (true) or waiting in
      queue for its turn (false).
    
    timestamp
      the timestamp of the last time we have seen request headers
      from this client.
    
    identifier
      this is a reference to the unique identifier.
    
    queuepos
      the position in queue of this client.
    
    negotiations
      the number of negotiations completed by this client.
    
    We model the queue with _SessionTrackerMixin, a mixin that contains
    the following fields:
    
    identifiers
      map an unique identifier to the related session object.
    
    queue
      keep track of the position in queue (fifo).
    
    The algorithm to decide whether a client is authorized or not to
    access restricted URIs is now very simple.  It has the permission
    (see check_request_headers()) if the session_active() method of
    the new mixin returns True.  This method returns True if the uuid
    of the new sessions is known _and_ the session is active.
    
    The negotiation algorithm now is as follows.  The function named
    session_negotiate of the _SessionTrackerMixin is now invoked.  This
    function returns the new session.  The new session will eventually
    have its active field set to True if it's one of the first three
    sessions in queue.  In this case it will be authorized to perform
    the test, and otherwise it will wait in queue.
    
    When a session uploads the results to /speedtest/collect, the
    function _speedtest_complete() is invoked, as before.  Now this
    function deletes the current session and then starts a new
    negotiation.
    
    Periodically the session_prune() method of the new mixin is
    invoked.  This method prunes all the sessions that have been
    idle for more than 30 seconds and returns True if it has
    removed at least one session.  In this case a new negotiation
    will take place.
    
    Whenever the queue changes (be it because we added a new session
    or we removed an old one or because we pruned stale sessions),
    the state of the queue is updated (see _do_update_queue() in the
    new mixin.)
    
    Still, this algorithm does not leverages the `connection lost` event,
    but this will be the subject of an upcoming patch.
  5. neubot/speedtest.py: Hack to avoid client disconnection

    bassosimone committed Nov 24, 2010
    http.Connection decrements self.left after each request, and closes
    the socket when such number becomes zero.  SpeedtestClient, that
    use http.Connection, might fail to speedtest because the Connection
    might close the socket when self.left becomes zero, e.g. after a
    given number of negotiations (or possibly even in the middle of
    the test.)
    
    In principle, we don't want that because this behavior prevents
    some clients from receiving the service.  And this patch provides
    a simple fix, forcing self.left to be always equal to 128.  But
    this hackish solution should be improved.
    
    BTW, second-thinking at this issue, I'm not sure it is a good idea
    to add another client to an already huge queue.  Perhaps it might be
    better to `drop` the connection with the incoming client in order
    to keep the queue length under control.  We'll third-think at that.
  6. neubot/speedtest.py: Cosmetics after refactoring into mixins

    bassosimone committed Nov 24, 2010
    Now that we have move the code to negotiate and the code to test into
    separate mixins we can apply some cosmetic changes to make the new code
    more pleasant.
    
    More in detail:
    
    - s/_do/do/ for now-public mixin methods,
    
    - remove unused self.config member of SpeedtestServer,
    
    - prefix server code block with explanatory comment.
  7. neubot/speedtest.py: Move negotiator server code in a separate mixin

    bassosimone committed Nov 24, 2010
    No functional change.  The rationale of this patch is to separate
    the code that deals with testing from the code that deals with the
    queue.
  8. neubot/speedtest.py: Move test server code into a separate mixin

    bassosimone committed Nov 24, 2010
    No functional change.  The rationale of this patch is to separate
    the code that deals with testing from the code that deals with the
    queue.
Commits on Dec 30, 2010
  1. neubot/state.py: Use "idle" instead of None to mark inactivity

    Roberto D'Auria authored and bassosimone committed Dec 17, 2010
    Comment from Simone:
      Roberto's take is that using "idle" the code would be more readable
      and I agree with him.
  2. rendezvous.py: Pass to state.py some timing information

    bassosimone committed Dec 17, 2010
    The upcoming changes in the new web user interface require use to
    export via /api/state also the time of the next rendezvous and the
    time since this instance of neubot has been started.
    
    Requested by Antonio.
  3. rendezvous.py: Give equal tratment to both test servers

    bassosimone committed Dec 16, 2010
    The purpose of this patch is to redirect half of the traffic to the
    additional speedtest server that runs a newer version of the code in
    order to stress-test the new version.
  4. rendezvous.py: Do not wait for a deterministic amount of time

    bassosimone committed Dec 16, 2010
    Suggested by Elias.
    
    The point here is that it is bad for clients to attempt the next
    rendezvous after a fixed amount of seconds.  Indeed this could synchronize
    all the neubot that are in queue at a certain amount of time if the
    speedtest server is restarted.
    
    However (if I have correctly understood the point made by Elias) it would
    be wrong to wait a random amount of time choosen uniformly in a given
    interval, because this will add noise to the process of sampling the network.
    
    To reduce the noise, it would be better to decide once and forever the
    random interval between each rendezvous.  And this is what this patch does.
  5. rendezvous.py: Notify state.py that connection failed

    Roberto D'Auria authored and bassosimone committed Dec 11, 2010
    Comment from Simone:
      This patch is particularly important because it implements a
      mechanism to add to /api/state output the information that something
      went wrong during the rendezvous (for example because the current
      network does not allow connecting to the address and port where the
      rendezvous server is listening).
    
      But this is just a mechanism and the obvious follow-up to this
      commit will be to amend the web user interface -and possibily the
      text user interface- to pass this information to the user.
  6. speedtest.py: Add -d command line option for -S mode

    bassosimone committed Nov 29, 2010
    Now that the speedtest server should run as a standalone process and
    not in the same process of the rendezvous server, it makes sense to
    amend the default behavior and drop privileges and background.  The old
    behavior (not dropping privileges and foreground) is still available
    via the new -d (debug) command line switch.
  7. rendezvous.py: On -S do not start a companion speedtest server

    bassosimone committed Nov 29, 2010
    The default behavior so far has been to run the rendezvous server
    and the speedtest server in the same unprivileged background process.
    The rendezvous server has always started the speedtest server as a
    part of its startup procedure.
    
    This commit modifies the rendezvous server so that it does not start
    the speedtest server automatically.  In other words, we want the speedtest
    server to live into its own process instead of sharing the same process
    with the rendezvous server.
    
    This change makes sense for a number of reasons.  The most important one
    is that we want to deploy an alternate speedtest server and we don't want
    to start up a rendezvous server just to deploy such alternate speedtest
    server.
Commits on Dec 29, 2010
  1. net/pollers.py: Add timestamping to PollerTask

    bassosimone committed Dec 17, 2010
    We need to add timestamping because we cannot be sure that the value in
    ticks is relative to EPOCH.  With windows, for example, that value is the
    number of -IIRC- microseconds since the program started.
  2. neubot/ui.py: Break cleanly out of the main loop

    bassosimone committed Dec 16, 2010
    Now that it's possible to break out of the main loop cleanly, let's do
    that in /api/exit.  No functional change, but this removes a very ugly XXX
    from the code base.
  3. net/pollers.py: Allow to break the loop cleanly

    bassosimone committed Dec 16, 2010
    At least the handler of /api/exit needs that.  Generally speaking it
    is a good idea for the program to be able to generate an event that
    causes the main loop to break (even if there is still something to do).
    
    A possible follow-up to this patch would be to break the loop if and
    only if the variable `again` becomes False (i.e. without even inspecting
    the value of readset and writeset).
  4. net/pollers.py: Don't use from..import for standard library

    bassosimone committed Dec 6, 2010
    Using from..import for standard library might be confusing for someone
    that already knows Python but is new to the project.  So, better to use
    plain import for standard library and from..import for neubot own
    modules.
    
    Many thanks to Alessio Palmero that pointed out this issue with the
    source code.
  5. net/pollers.py: Remove unused dispatch() method

    bassosimone committed Dec 6, 2010
    This method is not useful anymore and so remove it.
  6. net/pollers.py: move Stats and SimpleStats into utils.py

    bassosimone committed Dec 6, 2010
    SimpleStats and Stats are generic classes and therefore they should live
    in neubot/utils.py rather than in net/pollers.py.
  7. net/pollers.py: Always treat SystemExit like KeyboardInterrupt

    bassosimone committed Dec 6, 2010
    As promised, here's the patch that treats SystemExit like Keyboard-
    Interrupt for readable and writable.  The benefit is that we can
    invoke sys.exit() in /api/exit and possibly in other places and
    that will actually exit from the program (instead of catching the
    exception and merely spitting an error message out.)
  8. net/pollers.py: Surround stream.closing with try..except

    bassosimone committed Nov 29, 2010
    This patch augments neubot robustness because it's possible for the
    closing() function to fail and we don't want that to tear down the
    program.  BTW, note that this patch treats SystemExit the same way as
    KeyboardInterrupt.  A subsequent patch will modify the readable and
    writable methods to treat SystemExit like KeyboardInterrupt as well.
  9. net/pollers.py: Get rid of Poller.get_ticks() globally

    bassosimone committed Nov 28, 2010
    At some point I though it could make sense to pass the poller a
    function that tells the current time (for caching) but now it's clear
    that was not a good idea.  So zap it.
  10. net/pollers.py: Simplify task scheduler

    bassosimone committed Nov 28, 2010
    This patch applies the following changes:
    
    1. remove the periodic stuff that has been deprecated for a long
       time and should have been removed before,
    
    2. removes the unsched() method from the poller and instead
    
       2.1 add the unsched() method to the PollerTask
    
       2.2 let sched() return the new PollerTask
    
       this simplifies the code and removes a giant bug where only
       bound methods could be used reliably as tasks,
    
    3. modifies http/handlers.py to invoke the unsched() method of the
       PollerTask instead of employing the now-gone unsched() method of
       the poller.
  11. www/results.js: Make sure we plot all data points

    Roberto D'Auria authored and bassosimone committed Nov 29, 2010
    This patch ensures that we plot all data points.  The existing code
    was wrong because of an off-by-one error when preparing the arrays
    that hold the data points.
    
    The issue has been reported in the public mailing list:
    <http://www.neubot.org/pipermail/neubot/2010-November/000022.html>
    
    Signed-off-by: Simone Basso <bassosimone@gmail.com>
  12. www/results.js: oops upload and dowload labels were swapped

    bassosimone committed Nov 26, 2010
    Spotted by Roberto D'Auria and me.
  13. Finish improving copyright notice

    bassosimone committed Dec 5, 2010
    Add the name of the original author, affiliation, and the website
    URI <http://www.neubot.org/>.
    
    This patch applies the above changes to all the files that have
    not been changed yet.
  14. debian: Improve copyright notice

    bassosimone committed Dec 5, 2010
    Add the name of the original author, affiliation, and the website
    URI <http://www.neubot.org/>.
  15. bin: Improve copyright notice

    bassosimone committed Dec 5, 2010
    Add the name of the original author, affiliation, and the website
    URI <http://www.neubot.org/>.