OWTwitterGraph: Add a new widget #22

rokgomiscek · Dec 15, 2016

Issue

Added widget for creating network of twitter users.

Description of changes

Widget takes a list of twitter users and searches for their followers and friends.

Includes

Code changes
Tests
Documentation

kernc · Dec 16, 2016

kernc requested changes Dec 16, 2016

View changes

Looks good. I'd have someone with a twitter account have a look at it. @nikicc @astaric

kernc · Dec 16, 2016

doc/widgets/networktwitter.rst

+**Outputs**:
+
+-  **Followers Graph**
+    A graph of users that follow the selected users.


Missing empty line. Seems to render differently.

kernc · Dec 16, 2016

orangecontrib/network/widgets/OWNxTwitterGraph.py

+        CONSUMER_SECRET = CredentialManager('Twitter API Secret').key
+        OAUTH_TOKEN = ''
+        OAUTH_TOKEN_SECRET = ''
+        auth = twitter.oauth.OAuth(OAUTH_TOKEN, OAUTH_TOKEN_SECRET, CONSUMER_KEY, CONSUMER_SECRET)


This crashes with:

Traceback (most recent call last): File "/home/u/orange3/networks/orangecontrib/network/widgets/OWNxTwitterGraph.py", line 112, in fetch_users auth = twitter.oauth.OAuth(OAUTH_TOKEN, OAUTH_TOKEN_SECRET, CONSUMER_KEY, CONSUMER_SECRET) File "/home/u/venv/orange3/lib/python3.4/site-packages/twitter/oauth.py", line 97, in __init__ 'You must supply strings for token_secret and consumer_secret, not None.') twitter.auth.MissingCredentialsError: You must supply strings for token_secret and consumer_secret, not None.

Confirmed, have an error message "Please, provide an API key." here.

kernc · Dec 16, 2016

orangecontrib/network/widgets/OWNxTwitterGraph.py

+from Orange.widgets import widget, gui
+from Orange.data import Domain, StringVariable, Table
+import numpy as np
+from Orange.widgets.credentials import CredentialManager


Group imports more nicely?

kernc · Dec 16, 2016

orangecontrib/network/widgets/OWNxTwitterGraph.py

+import numpy as np
+from Orange.widgets.credentials import CredentialManager
+import orangecontrib.network as network
+from orangecontrib.text import twitter as txt_twitter


It's not common we do this. Add-ons are unreliable, at best. In particular, I don't think they make any API guarantees. If the sole reason for this is twitter API key validation, I assume, for lack of proper documentation, that the twitter lib should fail on OAuth handshake if the access token were incorrect?

This would probably fail if the user doesn't have Text add-on installed, too. Could/should we re-implement this here as well?

kernc · Dec 16, 2016

orangecontrib/network/widgets/OWNxTwitterGraph.py

+from PyQt4 import QtGui, QtCore
+from collections import defaultdict
+
+class OWNxTwitterGraph(widget.OWWidget):


Have two blank lines between globals.

kernc · Dec 16, 2016

orangecontrib/network/widgets/OWNxTwitterGraph.py

+        followers_graph = nx.Graph()
+        following_graph = nx.Graph()
+        all_users = nx.Graph()
+        users = self.users.toPlainText().split("\n")


Could this be split on commas as well? Or perhaps with regex on all characters that aren't on twitter username whitelist?

Wouldn't this make things all too complicated? Current implementation in Text is newline as well, so I vote for keeping this for now.

Complicated is a relative term.

users = re.split('[^\w_]+', self.users.toPlainText())

It doesn't work for me, it returns an error 'user not found'.

Must be something in the way you specify users. This just splits on characters not in (A-Z, a-z, 0-9, and _), might need adapting (to accept @ or skip empty strings or something like that). Point was: it's not necessarily complicated. 😄

kernc · Dec 16, 2016

orangecontrib/network/widgets/OWNxTwitterGraph.py

+
+        fwers_names=[]
+        fwing_names=[]
+        all_names=[]


Add whitespace.

kernc · Dec 16, 2016

orangecontrib/network/widgets/OWNxTwitterGraph.py

+        all_names=[]
+
+        for user in users:
+            result = t.users.show(screen_name=user)


Add include_entities=False.

kernc · Dec 16, 2016

orangecontrib/network/widgets/OWNxTwitterGraph.py

+            result = t.users.show(screen_name=user)
+            id = result["id"]
+
+            if id in mapping.keys():


if id in mapping:

kernc · Dec 16, 2016

orangecontrib/network/widgets/OWNxTwitterGraph.py

+            followers_graph.add_node(user_fwer_id)
+            following_graph.add_node(user_fwing_id)
+            all_users.add_node(user_all_id)
+


Could you high-level comment a bit what's going on in each block of this long method?

ajdapretnar · Dec 16, 2016

I tested this and I have a few requests. Did a checklist, probably @nikicc can help with many of them.

implement progress bar
if requests are too big, the widget crashes. Could we somehow check the number of queries and provide (max requests/number of queries) results? E.g. get me a 100 for Hillary Clinton and a 100 for Donald Trump, like sampling?
the widget indeed crashes without (proper) API credentials
the widget should (?) work with a data table as well in my opinion, so the user can check what is in his data. Network File supports this, Twitter Graph doesn't.
if Starting User is None, perhaps reformat the node label. Now it shows , user, which is strange. I'd prefer to have it show just user without the comma.
strange 'bug' (?) in the data. I queried jademsar, bzupan and OrangeDataMiner and it seems like the widget failed to recognize jademsar to be the same person (when connected as Following).

Otherwise, I love the improvements you've made, especially the labeling! ❤️

nikicc · Dec 16, 2016

nikicc reviewed Dec 16, 2016

View changes

I checked this quickly and this are my observations:

I tried for @ylecun, @RichardSocher, @geoff_hinton but it seems quite slow (~4min+ without any results). I also vote for some sort of progress bar.
When copying names directly from twitter some styling is retained. See the figure below:
The upper right bird in the icon is almost touching the circle border. Could we align the icon to allow some space between the circle border and the icon, like in other widgets?
When rate limit is exceeded (which happens very quickly for popular users) the code crashes with the following stack. I suggest we catch the exception and show an error in the widget.

Stack Trace:
Traceback (most recent call last):
  File "/Users/Niko/anaconda/envs/orange3pyqt5/lib/python3.5/site-packages/twitter/api.py", line 319, in _handle_response
    handle = urllib_request.urlopen(req, **kwargs)
  File "/Users/Niko/anaconda/envs/orange3pyqt5/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/Users/Niko/anaconda/envs/orange3pyqt5/lib/python3.5/urllib/request.py", line 472, in open
    response = meth(req, response)
  File "/Users/Niko/anaconda/envs/orange3pyqt5/lib/python3.5/urllib/request.py", line 582, in http_response
    'http', request, response, code, msg, hdrs)
  File "/Users/Niko/anaconda/envs/orange3pyqt5/lib/python3.5/urllib/request.py", line 510, in error
    return self._call_chain(*args)
  File "/Users/Niko/anaconda/envs/orange3pyqt5/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/Users/Niko/anaconda/envs/orange3pyqt5/lib/python3.5/urllib/request.py", line 590, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 429: Too Many Requests

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/Niko/Projects/orange3-network/orangecontrib/network/widgets/OWNxTwitterGraph.py", line 164, in fetch_users
    response = t.followers.ids(screen_name=user, cursor=cursor)
  File "/Users/Niko/anaconda/envs/orange3pyqt5/lib/python3.5/site-packages/twitter/api.py", line 312, in __call__
    return self._handle_response(req, uri, arg_data, _timeout)
  File "/Users/Niko/anaconda/envs/orange3pyqt5/lib/python3.5/site-packages/twitter/api.py", line 345, in _handle_response
    raise TwitterHTTPError(e, uri, self.format, arg_data)
twitter.api.TwitterHTTPError: Twitter sent status 429 for URL: 1.1/followers/ids.json using parameters: (cursor=1523886457862802540&oauth_consumer_key=vVAZ5ABqKoWe5nTn0EPvLPA2S&oauth_nonce=11861049535243809337&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1481899364&oauth_version=1.0&screen_name=%40ylecun&oauth_signature=2E5r1FuKO9F9qv358SNMbyjtIIs%3D)
details: {'errors': [{'code': 88, 'message': 'Rate limit exceeded'}]}

nikicc · Dec 16, 2016

orangecontrib/network/widgets/OWNxTwitterGraph.py

+        self.button = gui.button(box, self, "Create graph", self.fetch_users)
+
+    def open_key_dialog(self):
+        self.api_dlg.exec_()


If this is the only thing this method does, we can just provide self.api_dlg.exec_ as a callback to the button above and delete this method.

nikicc · Dec 16, 2016

orangecontrib/network/widgets/OWNxTwitterGraph.py

+
+            if id in mapping.keys():
+                user_fwer_id, user_fwing_id, user_all_id = mapping[id]
+                if not user_fwer_id:


Could we move the code of this and the following if into a method? They seem to be doing the same thing.

nikicc · Dec 16, 2016

orangecontrib/network/widgets/OWNxTwitterGraph.py

+            all_users.add_node(user_all_id)
+
+            cursor = -1
+            while cursor != 0:


Same here; could the code from this and the following while be moved to a common method?

ajdapretnar · Jan 11, 2017

@rokgomiscek Could we get this working in about two weeks so we can include it in the tutorial for DH or would it be better to wait (and skip it)?

rokgomiscek · Jan 11, 2017

I've fixed most of @kernc's requests, still working on your requests. I'll try to push the fixes on Friday.

janezd · Nov 7, 2018

@rokgomiscek, any plans to finish this?

rokgomiscek added some commits Dec 9, 2016

Add new widget

54cc161

Basic documentation for OWNxTwitterGraph

7059411

A test for OWNxTwitterGraph

de99203

PrimozGodec referenced this pull request Oct 27, 2017
Closed
[WIP] OWTwitterNetwork: Add a new widget #45

janezd closed this Nov 23, 2018

biolab/orange3-network

OWTwitterGraph: Add a new widget #22

OWTwitterGraph: Add a new widget #22

rokgomiscek commented Dec 15, 2016

rokgomiscek added some commits Dec 9, 2016

kernc requested changes Dec 16, 2016

View changes

kernc left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ajdapretnar commented Dec 16, 2016 •

edited by rokgomiscek

nikicc reviewed Dec 16, 2016

View changes

nikicc left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ajdapretnar commented Jan 11, 2017

This comment has been minimized.

rokgomiscek commented Jan 11, 2017

PrimozGodec referenced this pull request Oct 27, 2017

[WIP] OWTwitterNetwork: Add a new widget #45

This comment has been minimized.

janezd commented Nov 7, 2018

janezd closed this Nov 23, 2018

biolab/orange3-network

Join GitHub today

OWTwitterGraph: Add a new widget #22

Conversation

rokgomiscek commented Dec 15, 2016

Issue

Description of changes

Includes

rokgomiscek added some commits Dec 9, 2016

kernc requested changes Dec 16, 2016 View changes

kernc left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ajdapretnar commented Dec 16, 2016 • edited by rokgomiscek

nikicc reviewed Dec 16, 2016 View changes

nikicc left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ajdapretnar commented Jan 11, 2017

This comment has been minimized.

rokgomiscek commented Jan 11, 2017

PrimozGodec referenced this pull request Oct 27, 2017

This comment has been minimized.

janezd commented Nov 7, 2018

janezd closed this Nov 23, 2018

kernc requested changes Dec 16, 2016

View changes

ajdapretnar commented Dec 16, 2016 •

edited by rokgomiscek

nikicc reviewed Dec 16, 2016

View changes