Merge branch 'master' into android_tests_refactoring

CHANGELOG.md

@@ -13,39 +13,77 @@ at anytime.
   *
 
 ### Fixed
-  * Fixed handling cancelled blob and availability requests
-  * Fixed redundant blob requests to a peer
-  * Fixed https://github.com/lbryio/lbry/issues/923
-  * Fixed concurrent reflects opening too many files
-  * Fixed cases when reflecting would fail on error conditions
-  * Fixed deadlocks from occuring during blob writes
+  * Fixed slow startup for nodes with many lbry files
+  * Fixed setting the external ip on startup
+  * Fixed session startup not blocking on joining the dht
+  * Fixed several parsing bugs that prevented replacing dead dht contacts
+  * Fixed lbryid length validation
+  * Fixed an old print statement that polluted logs
+  * Fixed rpc id length for dht requests
 
 ### Deprecated
-  * Deprecated `blob_announce_all` JSONRPC command. Use `blob_announce` instead.
+  *
   *
 
 ### Changed
-  * Announcing by head blob is turned on by default
-  * Updated reflector server dns
+  * Use the first port available for the peer and dht ports, starting with the provided values (defaults of 3333 and 4444). This allows multiple lbrynet instances in a LAN with UPnP.
+  * Detect a UPnP redirect that didn't get cleaned up on a previous run and use it
+  * Bumped jsonschema requirement to 2.6.0
   * Moved tests into the lbrynet package.
   * Refactor some assert statements to accommodate the PYTHONOPTIMIZE flag set for Android.
+
+### Added
+  *
+  *
+
+### Removed
+  *
+  *
+
+
+## [0.17.0] - 2017-10-12
+### Fixed
+ * Fixed handling cancelled blob and availability requests
+ * Fixed redundant blob requests to a peer
+ * Fixed https://github.com/lbryio/lbry/issues/923
+ * Fixed concurrent reflects opening too many files
+ * Fixed cases when reflecting would fail on error conditions
+ * Fixed deadlocks from occuring during blob writes
+ * Fixed and updated`lbrynet.tests.dht`
+ * Fixed redundant dht id
+ * Fixed dht `ping` method
+ * Fixed raising remote exceptions in dht
+ * Fixed hanging delayedCall in dht node class
+ * Fixed logging error in dht when calling or receiving methods with no arguments
+ * Fixed IndexError in routingTable.findCloseNodes which would cause an empty list to be returned
+ * Fixed bug where last blob in a stream was not saved to blob manager
+
+### Deprecated
+ * Deprecated `blob_announce_all` JSONRPC command. Use `blob_announce` instead.
+
+### Changed
+ * Bumped `lbryschema` requirement to 0.0.12 [see changelog](https://github.com/lbryio/lbryschema/blob/master/CHANGELOG.md#0012---2017-10-12)
+ * Bumped `lbryum` requirement to 3.1.9 [see changelog](https://github.com/lbryio/lbryum/blob/master/CHANGELOG.md#319---2017-10-12)
+ * Announcing by head blob is turned on by default
+ * Updated reflector server dns
+ * Moved tests into the lbrynet package.
 
 ### Added
-  * Added WAL pragma to sqlite3
-  * Added unit tests for `BlobFile`
-  * Updated exchange rate tests for the lbry.io api
-  * Use `hashlib` for sha384 instead of `pycrypto`
-  * Use `cryptography` instead of `pycrypto` for blob encryption and decryption
-  * Use `cryptography` for PKCS7 instead of doing it manually
-  * Use `BytesIO` buffers instead of temp files when processing blobs
-  * Refactored and pruned blob related classes into `lbrynet.blobs`
-  * Changed several `assert`s to raise more useful errors
-  * Added ability for reflector to store stream information for head blob announce
-  * Added blob announcement information to API call status with session flag
+ * Added WAL pragma to sqlite3
+ * Added unit tests for `BlobFile`
+ * Updated exchange rate tests for the lbry.io api
+ * Use `hashlib` for sha384 instead of `pycrypto`
+ * Use `cryptography` instead of `pycrypto` for blob encryption and decryption
+ * Use `cryptography` for PKCS7 instead of doing it manually
+ * Use `BytesIO` buffers instead of temp files when processing blobs
+ * Refactored and pruned blob related classes into `lbrynet.blobs`
+ * Changed several `assert`s to raise more useful errors
+ * Added ability for reflector to store stream information for head blob announce
+ * Added blob announcement information to API call status with session flag
 
 ### Removed
-  * Removed `TempBlobFile`
-  * Removed unused `EncryptedFileOpener`
+ * Removed `TempBlobFile`
+ * Removed unused `EncryptedFileOpener`
 
 
 ## [0.16.3] - 2017-09-28

LICENSE

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2015-2016 LBRY Inc
+Copyright (c) 2015-2017 LBRY Inc
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the
 "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish,

docs/cli.md

@@ -759,6 +759,32 @@ Returns:
             resolvable
 ```
 
+## routing_table_get
+
+```text
+Get DHT routing information
+
+Usage:
+    routing_table_get
+
+Returns:
+    (dict) dictionary containing routing and contact information
+    {
+        "buckets": {
+            <bucket index>: [
+                {
+                    "address": (str) peer address,
+                    "node_id": (str) peer node id,
+                    "blobs": (list) blob hashes announced by peer
+                }
+            ]
+        },
+        "contacts": (list) contact node ids,
+        "blob_hashes": (list) all of the blob hashes stored by peers in the list of buckets,
+        "node_id": (str) the local dht node id
+    }
+```
+
 ## settings_get
 
 ```text
@@ -857,6 +883,8 @@ Returns:
             'session_status': {
                 'managed_blobs': count of blobs in the blob manager,
                 'managed_streams': count of streams in the file manager
+                'announce_queue_size': number of blobs currently queued to be announced
+                'should_announce_blobs': number of blobs that should be announced
             }
 
         If given the dht status option:

lbrynet/__init__.py

@@ -1,6 +1,6 @@
 import logging
 
-__version__ = "0.17.0rc13"
+__version__ = "0.17.1rc4"
 version = tuple(__version__.split('.'))
 
 logging.getLogger(__name__).addHandler(logging.NullHandler())

lbrynet/blob/blob_file.py

@@ -168,7 +168,6 @@ def close_read_handle(self, file_handle):
         """
         if file_handle is not None:
             file_handle.close()
-            self.readers -= 1
 
     def reader_finished(self, reader):
         self.readers -= 1

lbrynet/core/Session.py

@@ -37,18 +37,18 @@ class Session(object):
     """
 
     def __init__(self, blob_data_payment_rate, db_dir=None,
-                 lbryid=None, peer_manager=None, dht_node_port=None,
+                 node_id=None, peer_manager=None, dht_node_port=None,
                  known_dht_nodes=None, peer_finder=None,
                  hash_announcer=None, blob_dir=None,
                  blob_manager=None, peer_port=None, use_upnp=True,
                  rate_limiter=None, wallet=None,
                  dht_node_class=node.Node, blob_tracker_class=None,
-                 payment_rate_manager_class=None, is_generous=True):
+                 payment_rate_manager_class=None, is_generous=True, external_ip=None):
         """@param blob_data_payment_rate: The default payment rate for blob data
 
         @param db_dir: The directory in which levelDB files should be stored
 
-        @param lbryid: The unique ID of this node
+        @param node_id: The unique ID of this node
 
         @param peer_manager: An object which keeps track of all known
             peers. If None, a PeerManager will be created
@@ -101,7 +101,7 @@ def __init__(self, blob_data_payment_rate, db_dir=None,
         """
         self.db_dir = db_dir
 
-        self.lbryid = lbryid
+        self.node_id = node_id
 
         self.peer_manager = peer_manager
 
@@ -124,7 +124,7 @@ def __init__(self, blob_data_payment_rate, db_dir=None,
 
         self.rate_limiter = rate_limiter
 
-        self.external_ip = '127.0.0.1'
+        self.external_ip = external_ip
 
         self.upnp_redirects = []
 
@@ -142,8 +142,8 @@ def setup(self):
 
         log.debug("Starting session.")
 
-        if self.lbryid is None:
-            self.lbryid = generate_id()
+        if self.node_id is None:
+            self.node_id = generate_id()
 
         if self.wallet is None:
             from lbrynet.core.PTCWallet import PTCWallet
@@ -193,6 +193,31 @@ def _try_upnp(self):
 
         log.debug("In _try_upnp")
 
+        def get_free_port(upnp, port, protocol):
+            # returns an existing mapping if it exists
+            mapping = upnp.getspecificportmapping(port, protocol)
+            if not mapping:
+                return port
+            if upnp.lanaddr == mapping[0]:
+                return mapping
+            return get_free_port(upnp, port + 1, protocol)
+
+        def get_port_mapping(upnp, internal_port, protocol, description):
+            # try to map to the requested port, if there is already a mapping use the next external
+            # port available
+            if protocol not in ['UDP', 'TCP']:
+                raise Exception("invalid protocol")
+            external_port = get_free_port(upnp, internal_port, protocol)
+            if isinstance(external_port, tuple):
+                log.info("Found existing UPnP redirect %s:%i (%s) to %s:%i, using it",
+                         self.external_ip, external_port[1], protocol, upnp.lanaddr, internal_port)
+                return external_port[1], protocol
+            upnp.addportmapping(external_port, protocol, upnp.lanaddr, internal_port,
+                                description, '')
+            log.info("Set UPnP redirect %s:%i (%s) to %s:%i", self.external_ip, external_port,
+                     protocol, upnp.lanaddr, internal_port)
+            return external_port, protocol
+
         def threaded_try_upnp():
             if self.use_upnp is False:
                 log.debug("Not using upnp")
@@ -202,40 +227,15 @@ def threaded_try_upnp():
             if num_devices_found > 0:
                 u.selectigd()
                 external_ip = u.externalipaddress()
-                if external_ip != '0.0.0.0':
+                if external_ip != '0.0.0.0' and not self.external_ip:
+                    # best not to rely on this external ip, the router can be behind layers of NATs
                     self.external_ip = external_ip
-                if self.peer_port is not None:
-                    if u.getspecificportmapping(self.peer_port, 'TCP') is None:
-                        u.addportmapping(
-                            self.peer_port, 'TCP', u.lanaddr, self.peer_port,
-                            'LBRY peer port', '')
-                        self.upnp_redirects.append((self.peer_port, 'TCP'))
-                        log.info("Set UPnP redirect for TCP port %d", self.peer_port)
-                    else:
-                        # see comment below
-                        log.warning("UPnP redirect already set for TCP port %d", self.peer_port)
-                        self.upnp_redirects.append((self.peer_port, 'TCP'))
-                if self.dht_node_port is not None:
-                    if u.getspecificportmapping(self.dht_node_port, 'UDP') is None:
-                        u.addportmapping(
-                            self.dht_node_port, 'UDP', u.lanaddr, self.dht_node_port,
-                            'LBRY DHT port', '')
-                        self.upnp_redirects.append((self.dht_node_port, 'UDP'))
-                        log.info("Set UPnP redirect for UDP port %d", self.dht_node_port)
-                    else:
-                        # TODO: check that the existing redirect was
-                        # put up by an old lbrynet session before
-                        # grabbing it if such a disconnected redirect
-                        # exists, then upnp won't work unless the
-                        # redirect is appended or is torn down and set
-                        # back up. a bad shutdown of lbrynet could
-                        # leave such a redirect up and cause problems
-                        # on the next start.  this could be
-                        # problematic if a previous lbrynet session
-                        # didn't make the redirect, and it was made by
-                        # another application
-                        log.warning("UPnP redirect already set for UDP port %d", self.dht_node_port)
-                        self.upnp_redirects.append((self.dht_node_port, 'UDP'))
+                if self.peer_port:
+                    self.upnp_redirects.append(get_port_mapping(u, self.peer_port, 'TCP',
+                                                                'LBRY peer port'))
+                if self.dht_node_port:
+                    self.upnp_redirects.append(get_port_mapping(u, self.dht_node_port, 'UDP',
+                                                                'LBRY DHT port'))
                 return True
             return False
 
@@ -260,8 +260,7 @@ def join_resolved_addresses(result):
                     addresses.append(value)
             return addresses
 
-        def start_dht(addresses):
-            self.dht_node.joinNetwork(addresses)
+        def start_dht(join_network_result):
             self.peer_finder.run_manage_loop()
             self.hash_announcer.run_manage_loop()
             return True
@@ -274,7 +273,7 @@ def start_dht(addresses):
 
         self.dht_node = self.dht_node_class(
             udpPort=self.dht_node_port,
-            lbryid=self.lbryid,
+            node_id=self.node_id,
             externalIP=self.external_ip
         )
         self.peer_finder = DHTPeerFinder(self.dht_node, self.peer_manager)
@@ -283,6 +282,7 @@ def start_dht(addresses):
 
         dl = defer.DeferredList(ds)
         dl.addCallback(join_resolved_addresses)
+        dl.addCallback(self.dht_node.joinNetwork)
         dl.addCallback(start_dht)
         return dl
 

lbrynet/core/client/ClientProtocol.py

@@ -140,9 +140,8 @@ def send_request_or_close(do_request):
                 self._send_request_message(request_msg)
             else:
                 # The connection manager has indicated that this connection should be terminated
-                log.info(
-                    "Closing the connection to %s due to having no further requests to send",
-                    self.peer)
+                log.debug("Closing the connection to %s due to having no further requests to send",
+                          self.peer)
                 self.peer.report_success()
                 self.transport.loseConnection()
         d = self._connection_manager.get_next_request(self.peer, self)

lbrynet/core/client/DHTPeerFinder.py

@@ -46,7 +46,10 @@ def _trigger_timeout():
         if timeout is not None:
             reactor.callLater(timeout, _trigger_timeout)
 
-        peer_list = yield finished_deferred
+        try:
+            peer_list = yield finished_deferred
+        except defer.CancelledError:
+            peer_list = []
 
         peers = set(peer_list)
         good_peers = []