Adding support for hashtag or keyword search #1518

Closed
wants to merge 43 commits into
from

Projects

None yet

5 participants

@ecucurella

Introduction

Twitter hashtags search collects all tweets from Twitter API 1.1 tweets/search
that contain keyword or hashtag specified related to a twitter account.

User interface:

  • A twitter account can add hashtags or keywords to be searched
  • A twitter account can delete hashtags or keywords from being searched
  • View how many hashtags or keywords are being searched and how many tweets have been saved for each one
  • View predefined search of hashtag or keyword

Screenshot for Twitter Plugin settings:
ThinkUp_TwitterSettings

Screenshot for Twitter Plugin Search settings:
ThinkUp_TwitterSearchTweetsSettings

Screenshot for new options in statusbar search:
ThinkUp_StatusbarSearch

Screenshot for viewing predefined search of hashtag or keyword:
ThinkUp_ResultsForSearch

Twitter Crawler

We have added tweets search for all hashtag or keyword related to every instance

POST API

We have added a new type for POST API named hashtag_posts to get posts that contain a hashtag or keyword.

Changes

Docs

  • Added documentation for new Exception in POST API HashtagNotFoundException
  • Added documentation for new type of POST API hashtag_posts
  • Added documentation for Twitter hashtags or Keyword search
  • Modified some index to add new documentation

Tests

  • Added TestOfHashtagPostMySQLDAO
  • Added TestOfInstanceHashtagMySQLDAO
  • Modified TestOfAccountConfigurationController, new test
  • Modified TestOfDAOFactory, modified assertEqual for increase number of tables in database
  • Modified TestOfHashtagMySQLDAO, added new test and modified some test with change of buildata
  • Modified TestOfInstaller, adding new table instances_hashtags
  • Modified TestOfInstallerMySQLDAO, modified assertEqual for increase number of tables in database
  • Modified TestOfLinkMySQLDAO, new test
  • Modified TestOfPostAPIController, new test
  • Modified TestOfPostMySQLDAO, new test and modified some test with change of buildata
  • Modified TestOfReporter, modification to pass test
  • Modified TestOfUserMySQLDAO, new test
  • Modified WebTestOfDeleteInstance, set configuration for proxy options
  • Modified WebTestOfUpgradeDatabase, set configuration for proxy options
  • Modified TestOfSearchController, new test

Webapp

  • Modified config.sample.inc.php with new commented values in TEST MODE for invalid_host, requires_proxy and proxy

lib

  • Modified class DAOFactory adding new pair of key, value in DAO mapping
  • Modified class Installer to solve one test fail

controller

  • Modified class AccountConfigurationController to be able to:
    ** add new hashtag or keyword search
    ** delete a hashtag or keyword search
    ** delete all hashtag or keyword searches from an instance when this instance is deleted
    ** render configuration view for hashtag or keyword search
  • Modified class PostAPICOntroller, adding a new type named hashtag_posts to get posts with a hashtag or keyword
  • Modified class SearchController, adding new case for viewing tweets from hashtag or keyword search

dao

  • Added class HashtagPostMySQLDAO that extends from HashtagMySQLDAO and overwrites insertHashtag
  • Added class InstanceHashtagMySQLDAO that implement interface InstanceHashtagDAO
  • Added interface InstanceHashtagDAO
  • Modified class HashtagMySQLDAO, add new public functions
  • Modified interface HashtagDAO
  • Modified class LinkMySQLDAO, add new public function
  • Modified interface LinkDAO
  • Modified class PostMySQLDAO, add new public function
  • Modified interface PostDAO
  • Modified class UserMySQLDAO, add new public function
  • Modified interface UserDAO

exceptions

  • Added class HashtagNotFoundException that extends from Exception

model

  • Added class Hashtag
  • Added class HashtagPost
  • Added class InstanceHashtag

install

sql

  • Generated with script migratedb file build-db_mysql.sql
mysql_migrations
  • Added new sql file to create table tu_instances_hashtags

plugins

twitter

controller
  • Added class TwitterPluginHashtagConfigurationController for view configuration of hashtag or keyword search
  • Modified class TwitterAuthController to add configuration of proxy values
  • Modified class TwitterPluginConfigurationController to add options for configure proxy settings in Twitter Plugin Configuration
extlib
twitteroauth
  • Modified twitteroauth http function to set options for proxy in curl
model
  • Modified class CrawlerTwitterAPIAccessorOAuth to:
    ** add proxy settings
    ** add search in resources string to initialize Rate limit
  • Modified class TwitterAPIAccessorOAuth to:
    ** add proxy settings
    ** add search/tweets in api_method array
  • add function parseJSONTweetsFromSearch
  • Modified class TwitterCrawler, added function fetchInstanceHashtagTweets
  • Modified class TwitterOAuthThinkUp, added setProxy function
  • Modified class TwitterPlugin, modified crawl function to call fetchInstanceHashtagTweets for each instance_hashtag relation
tests
  • Modified TestOfTwitterAPIAccessorOAuth, new test
  • Modified TestOfTwitterAuthController for proxy settings
  • Modified TestOfTwitterCrawler, new tests
  • Modified TestOfTwitterPlugin for proxy settings
testdata
  • Add application_rate_limit_status.json-resources=account,statuses,users,followers,lists,friends,favorites,friendships,application,search
  • Add json/search_tweets.json
  • Add testofcrawlertwitterapiaccessoroauth/testinitializeendpointratelimits/application_rate_limit_status.json-resources=account,statuses,users,followers,lists,friends,favorites,friendships,application,search
view
  • Add twitter.account.hashtag.tpl for hashtag or keyword search settings
  • Modified twitter.hashtag.tpl for adding button to configure hashtag or keyword search settings
  • Modified _statusbar.tpl adding new options in dropdown menu
  • Modified search.tpl for new search

Powered by Televisió de Catalunya / Universitat Illes Balears - SOCIALMEDIA PROJECT - CEN-20101037 - This contribution has been kindly sponsored by the Centro para el Desarrollo Tecnológico Industrial within the Programa de Investigación Nacional Español CENIT. - http://www.cenitsocialmedia.es/

ginatrapani and others added some commits Feb 25, 2013
@ginatrapani ginatrapani Fix insights stream error when user is not logged in 4f766f3
@ginatrapani ginatrapani Version 2.0-beta.4 9e154a7
@ginatrapani ginatrapani 1.3.1 docs 741c7e8
@ginatrapani ginatrapani Add TestOfTwitterPlugin 8044164
@ginatrapani ginatrapani Fix TestOfFavoritePostMySQLDAO to account for short Februrary 4dda4fb
@ginatrapani ginatrapani Improve Twitter plugin crawl logging
Don't show repetitive status messages about tweet count
Format numbers, add periods
3f97776
@ginatrapani ginatrapani Fix TestOfGroupMemberMySQLDAO when it runs between midnight and 1am a1f4c67
@anildash anildash Search refinements and style cleanup
Enable beginning of Facebook Graph Search-style UI
Convert search refinement interface to native Bootstrap dropdown
Fix mobile styling for new graph search UI
Begin support for name: searching
Extensive cleanup on styling in Plugins/Manage settings configuration page
Enable default search value if no menu option is selected.
Hide search form on mobile, closes #1507
Update tests for new app copy and Settings cleanup
440e9dd
root Fix for bug #1497. e873fd4
@ginatrapani ginatrapani Twitter plugin: update instance, cache dashboard modules, and report …
…version even if an Exception was thrown
e524834
@ginatrapani ginatrapani Another TestOfFavoritePostMySQLDAO fix e0fb12a
@jgeurts jgeurts Fix register & forgot password links on login page b2a4204
@ginatrapani ginatrapani Exclude crawler directory from Utils siteRootPath calculation 7d48e99
@ginatrapani ginatrapani Twitter plugin throw exception if there are no OAuth tokens 0231a25
@ginatrapani ginatrapani Activate Insights Generator plugin by default 45c57de
@ginatrapani ginatrapani Insights Generator activation test fixes 20edb90
@ginatrapani ginatrapani Insights Generator activation test fixes 5e6b4a6
@ginatrapani ginatrapani Account for glob() incompatibility on Travis 3380b5d
@ginatrapani ginatrapani Fix PHP Notice, closes #1503 1475d9f
@ginatrapani ginatrapani Fix TestOfPluginOptionController failure b146ed5
@ginatrapani ginatrapani Fix PHP Notices in TwitterCrawler closes #1504 e9b35c7
@ginatrapani ginatrapani Fix PHP Notice: Undefined index: HTTP_HOST 95f9cd1
@ginatrapani ginatrapani Fix PHP Notice: Undefined offset: 0 61c4427
@ginatrapani ginatrapani Avoid division by zero error in style stats insight 131ce57
@ginatrapani ginatrapani Bugfix for crawler poop-out on follow 404 b424228
@ginatrapani ginatrapani Show number of search results on first page, closes #1510 2128e23
@ginatrapani ginatrapani Reduce number of insights per page to 20 65357a3
@ecucurella ecucurella Adding support for hashtag or keyword search
Adding support for hashtag or keyword search

Powered by Televisió de Catalunya / Universitat Illes Balears -
SOCIALMEDIA PROJECT - CEN-20101037 - This contribution has been kindly
sponsored by the Centro para el Desarrollo Tecnológico Industrial within
the Programa de Investigación Nacional Español CENIT. -
http://www.cenitsocialmedia.es/
79fd660
@ecucurella ecucurella Add insights for hashtag or keyword posts
Added new case on SearchController for hashtag or keyword posts
Added new options on _statusbar.tpl and on search.tpl
Added tests for changes
Code changed for curly braces coding style

Powered by Televisió de Catalunya / Universitat Illes Balears -
SOCIALMEDIA PROJECT - CEN-20101037 - This contribution has been kindly
sponsored by the Centro para el Desarrollo Tecnológico Industrial within
the Programa de Investigación Nacional Español CENIT. -
http://www.cenitsocialmedia.es/
2fedd49
@ecucurella

In commit "Add insights for hashtag or keyword posts" we have added to view predefined search of hashtag or keyword
Also we have checked curly braces from coding guidelines
Hope it helps

@ginatrapani
Member

Thanks very much for all your hard work on this!

Before I begin code review, may I ask a favor? It looks like your commits didn't rebase cleanly because of e873fd4, so we're seeing all those other unrelated changes here in this pull request. Could you remove that commit and rebase on the current state of the master branch?

Thanks so much.

@ecucurella

OK, no problem.
I will revert changes in search.tpl for this commit and the addition of t parameter to distinguish between hashtag and keyword.
Then I will rebase on current state of the master branch.
Comment when finished

anildash and others added some commits Mar 26, 2013
@anildash anildash Clean up status bar to not show insights if no instances are set up, …
…and to disable smart search autocomplete.
63cc311
@anildash anildash Replace footer images with font icons for social links 79c22c5
@ginatrapani ginatrapani Collapse redundant insights with low emphasis in stream 8cdc68f
@ginatrapani ginatrapani Roll back 94e9050 f6373f0
@ginatrapani ginatrapani Avoid PHP Warning: json_encode(): Invalid UTF-8 sequence in argument 0fb6589
@ginatrapani ginatrapani Fix set private/public button success messages in plugin settings cfb8f94
@ginatrapani ginatrapani Handle Twitter authorization on plugin page, don't require another click a8c0e2e
@ginatrapani ginatrapani Fix hot post encoding on PHP 5.4 6527295
@anildash anildash Remove nav link borders on mobile UAs. 06ade0b
@ginatrapani ginatrapani Test encoding on PHP 5.4 2a824ee
@anildash anildash Convert installation to Bootstrap design, refactor client-side valida…
…tion to be Bootstrap/HTML5 compatible, update style of backup/export pages

* Upgraded detect_timezone.js to jstz-1.0.4.min.js, to bring in 3 years' worth of updates
* Replace older jquery.validate.js in installation with jqBootstrapValidation.js (older validate still used in registration flow)
* Update necessary tests for copy changes in installation screens
cde30c6
@ginatrapani ginatrapani Version 2.0-beta.5 f198216
@ecucurella ecucurella Adding support for hashtag or keyword search
Adding support for hashtag or keyword search

Powered by Televisió de Catalunya / Universitat Illes Balears -
SOCIALMEDIA PROJECT - CEN-20101037 - This contribution has been kindly
sponsored by the Centro para el Desarrollo Tecnológico Industrial within
the Programa de Investigación Nacional Español CENIT. -
http://www.cenitsocialmedia.es/
a6fb0c1
@ecucurella ecucurella Revert commit "Fix for bug #1497" and own changes
Revert commit "Fix for bug #1497" and own changes in search.tpl

Powered by Televisió de Catalunya / Universitat Illes Balears -
SOCIALMEDIA PROJECT - CEN-20101037 - This contribution has been kindly
sponsored by the Centro para el Desarrollo Tecnológico Industrial within
the Programa de Investigación Nacional Español CENIT. -
http://www.cenitsocialmedia.es/
21b16ae
@ecucurella

Gina,

I have revert commit you comented.
Also I have rebase on the current state of the master branch.

But I have a doubt in webapp/_lib/view/search.tpl about two parameters left (c and new one t)

... {if $smarty.get.c}c={$smarty.get.c}&{/if}{if $smarty.get.t}t={$smarty.get.t}&{/if} ...

Hope it helps,

Eduard

@danigiri
danigiri commented Apr 3, 2013

+1
Thanks guys for your efforts (happy to have commissioned Eduard's work).

@ginatrapani
Member

Hi there--just wanted to let you know I've rebased and squashed and I'm currently code-reviewing the changes in this branch:
https://github.com/ginatrapani/ThinkUp/tree/1518_hashtag-search

The hashtag search is working great on my dev workstation! But this pull request includes changes to the application outside of the hashtag search, like proxy support, curly braces, and maybe more I'm not seeing yet, so it will take me some time to do a full review. I hope to have more time to review further this weekend.

Thanks for your patience.

@ginatrapani
Member

Update: I made some progress reviewing these changes. Overall they are well-structured and demonstrate a deep understanding of ThinkUp's architecture and style. Nice work. I especially appreciate the thorough testing.

I've already merged the code style corrections (curly braces) into the master branch. From there, I broke this feature down into its parts: DAO changes, plugin settings UI, Twitter crawler changes, API change, and search changes.

Code which has passed review so far (DAO changes):
https://github.com/ginatrapani/ThinkUp/tree/1518-passed-review

What I'm working on now (plugin settings UI):
https://github.com/ginatrapani/ThinkUp/commits/1518-work-in-progress

This needs some work; I listed the TODO's in this commit:
2795f4d

And what's left in the queue (crawler, search, and API changes):
https://github.com/ginatrapani/ThinkUp/commits/1518-needs-review

I removed the code which adds proxy server support because it's out of scope. (Also, it's unfinished and undocumented, at one point it includes a hardcoded proxy IP address, which we do not want):
https://github.com/ginatrapani/ThinkUp/commits/1518_proxy-support

I'm going to take a break to work on some of the other smaller pull requests and will come back to this later this week. Thanks for your patience.

@ecucurella

Hi Gina, I appreciate your comments, feels good.
Do you want me to make TODO's ?
How do you want to proceed ?

@ginatrapani
Member

Hi Eduard,

I'm going to work on this more this weekend and I'll let you know if I have questions. My ambitious goal is to merge by Sunday night. I think we're close!

Thanks again,
Gina

@ginatrapani ginatrapani added a commit that closed this pull request Apr 23, 2013
@ecucurella ecucurella Twitter saved searches
* Add interface for saving keyword/hashtag searches on Twitter
* Fetch search results from Twitter.com during crawl
* Add saved searches to ThinkUp's search dropdown

Closes #1518
70fe6d8
@ginatrapani
Member

Hi Eduard,

I just merged Twitter saved search feature into master. Congratulations!

One general change I made is to refer to the feature as saved searches in the user interface, instead of hashtags. Since there's really no functional difference between a hashtag and a keyword, I didn't want the interface to imply it was hashtag-specific.

I also refactored how ThinkUp's search dropdown lets users access saved searches. Instead of differentiating between keywords and hashtags, I just listed each saved search and let the user search within those sets of tweets.

The only code I did not merge was the proxy-support code (as I mentioned earlier), and also the Post API changes. The API changes required the user pass a hashtag_id on the query string, and as a general rule we don't expose internal IDs to users. To make saved searches available via the API, we need to refactor that to take the keyword and network instead of the hashtag ID.

Let me know if you have any questions. This was a massive commit so I do expect to tweak/fix things as we test further.

Thanks for all your work on this and @danigiri thank you for facilitating.
Gina

@ecucurella

Hi,

Thank you Gina, we are proud of participating in ThinkUp project.

Changes you have made are logical.

  • We needed proxy support for test instances but not for production instance, so maybe it's ok without it
  • Hashtag and keyword differentiation was difficult to take, we had doubts about it, so ok too.

Perhaps, the only unmerged code I am not agree is about Post API:

  • We could delete hashtag_id parameter and preserve hashtag_name parameter
  • We need this functionality for our outside developments

Best regards,

Eduard

@ginatrapani
Member

Eduard,

I completely understand about the API changes. Can you refactor the API to take keyword and network on the query string, and then use those to retrieve the ID from the database? If so, send me a pull request for those changes (rebased on my current master branch) and I'll review and merge.

I'm also open to merging proxy support, it just has to be documented and cleaned up. If you have the time to do that, send me a separate pull request and we'll work through it.

Best,
Gina

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment