Singleton and Member cache cleanup

[boudewijn-tribler]

All Singleton methods are no longer available:

• dispersy.has_instance
• dispersy.get_instance
• dispersy.del_instance

The Dispersy instance is available directly from the LaunchManyCore,
or though the session instance (session.lm.dispersy).

Also, DummyMember and Member instances should only be created through
Dispersy using:

• dispersy.get_temorary_member_from_id (returns a DummyMember)
• dispersy.get_new_member
• dispersy.get_member
• dispersy.get_members_from_id
• dispersy.get_member_from_database_id

[boudewijn-tribler]

There is still some work that needs to be discussed. Namely, where the callback should start/stop.

Up until now the callback was created and started by the LaunchManyCore. Hence, the callback should also be stopped by the LaunchManyCore. If we want, we can move both the start and stop to Dispersy. The same thing goes for the Endpoint, this is currently created in the LaunchManyCore and passed to Dispersy as a parameter.

Whatever we decide, we need to take into account that Dispersy is also started from dispersy/tool/main.py (and possibly other locations). When running from main.py dispersy should not call callback.start/stop since this would create a new thread, but instead run callback.loop on the main thread. Finally, StandaloneEndpoint has a start/stop that is called from main.py.

[NielsZeilemaker]

I'm trying to reply to the comment of Boudewijn.
I agree, if the Launchmanycore starts the callbackthread, it should stop it.
Also calling dispersy.stop in order to clean some stuffs should be handled by launchmanycore.

[boudewijn-tribler]

After a chat with Niels we agree on the following:

• we will create Callback and Endpoint outside of Dispersy and pass them as parameters
• we will add a start method to Dispersy
• Dispersy.start will...
  1. start all registered communities (previously done in LaunchManyCore),
  2. start Callback, and
  3. start Endpoint.

Hence, we need to add start and stop methods to all Endpoint classes, even though only StandaloneEnpoint actualy uses it, and we will need to add a NoThreadCallback class that doesn't start a thread to use within dispersy/tool/main.py.

[boudewijn-tribler]

I am not seeing any channels so something is broken, unfortunately I am not getting any errors. I had the same thing a few days ago, and Egbert could see errors...

[boudewijn-tribler]

I have seen each test in http://jenkins.tribler.org/jenkins/job/Test_tribler_full-ui-run_stop working on my own PC at least once. test_libtorrent_download.py:test_stopresumedelete seems the most error prone, perhaps because it relies on external resources such as torrent files and peers. However, this test inconsistency is unrelated to this pull request.

I see no more issues holding this merge back. Please comment if you feel differently.

[NielsZeilemaker]

The only strange thing I see is that the dht cannot bind on the socket.
It's reporting that its in use.
On which port is Dispersy binding? Is that causing a conflict?

2013/4/15 Boudewijn notifications@github.com

I have seen each test in
http://jenkins.tribler.org/jenkins/job/Test_tribler_full-ui-run_stopworking on my own PC at least once.
test_libtorrent_download.py:test_stopresumedelete seems the most error
prone, perhaps because it relies on external resources such as torrent
files and peers. However, this test inconsistency is unrelated to this pull
request.

I see no more issues holding this merge back. Please comment if you feel
differently.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/79#issuecomment-16382456
.

[boudewijn-tribler]

lmc: Dispersy is listening on port 7759

As far as I know the Dispersy port number hasn't changed for a long time. But I recall that a relatively recent change to the DHT port number used the default port - 1... and DEFAULTPORT=7760.

I'm trying to find the change to confirm...

[boudewijn-tribler]

Confirmed. It wasn't a problem before because Dispersy would add 1 to the port and try again, but since we now call Dispersy start earlier it is becoming the dht's problem. And the dht can't handle it.

Ive change it to using port 7757 (7758 is already taken by swift)

[NielsZeilemaker]

Great, maybe run the ui tests again to check if the problem is resolved.

2013/4/15 Boudewijn notifications@github.com

Confirmed. It wasn't a problem before because Dispersy would add 1 to the
port and try again, but since we now call Dispersy start earlier it is
becoming the dht's problem. And the dht can't handle it.

Ive change it to using port 7757 (7758 is already taken by swift)

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/79#issuecomment-16385240
.

[boudewijn-tribler]

No, didn't resolve the existing issue. I suggest we attempt to solve below bug in its own issue, since this is unrelated to this pull request.

ERROR: test_downloadfrommagnet (Tribler.Test.test_libtorrent_download.TestLibtorrentDownload)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/workspace/Test_tribler_full-ui-run_stop/tribler/Tribler/Test/test_gui_as_server.py", line 153, in tearDown
    assert boolean, reason
AssertionError: no download progress

[boudewijn-tribler]

I've fixed a bug in the cache db, see boudewijn-tribler@0a3b2d4. The database singleton was never removed/closed at tribler shutdown, and the testcases kept using the same database cursor over and over. This solves an exception that occurs when the final call to commitNow doesn't cause a commit because there is nothing too commit.

Anyway. The test AssertionError: no download progress still occurs every now and then. It works just fine (sometimes) on my laptop, and it consistently fails running on Jenkins.

[NielsZeilemaker]

Gui test seems to be working now, I'll ask Egbert to look into the
duplicate download exception.
Niels

2013/4/16 Boudewijn notifications@github.com

I've fixed a bug in the cache db, see boudewijn-tribler/tribler@0a3b2d4boudewijn-tribler@0a3b2d4.
The database singleton was never removed/closed at tribler shutdown, and
the testcases kept using the same database cursor over and over. This
solves an exception that occurs when the final call to commitNow doesn't
cause a commit because there is nothing too commit.

Anyway. The test AssertionError: no download progress still occurs every
now and then. It works just fine (sometimes) on my laptop, and it
consistently fails running on Jenkins.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/79#issuecomment-16434693
.

[boudewijn-tribler]

Unless anyone feels the duplicate download is related to this pull request, I suggest we discuss improvement/cleanup to sqlitecachedb.py on a different issue. Also, performing this merge will allow us to continue with the dprint modifications without causes tons of merge problems.

[NielsZeilemaker]

It's not, looking at this (
http://jenkins.tribler.org/jenkins/job/Test_tribler_full-ui-run_stop/18/artifact/tribler/output//Screenshot-09.png)
screenshot we can see that libtorrent is unable to fetch the .torrent from
the dht.
A clear libtorrent issue.

2013/4/16 Boudewijn notifications@github.com

Unless anyone feels the duplicate download is related to this pull
request, I suggest we discuss improvement/cleanup to sqlitecachedb.py on a
different issue. Also, performing this merge will allow us to continue with
the dprint modifications without causes tons of merge problems.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/79#issuecomment-16435445
.

[whirm]

This shouldn't be here, we are going to do both Dispersy and Tribler merge at the same time, right?

[boudewijn-tribler]

Yea, you are right. but when I remove it the Jenkins task will point to the wrong repro. So it should be changed just before the merge.

Please enlighten me if there was another (better) way to do this!

[whirm]

I can't think of any, right now :)