Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Submit your bounty entry here #1

Open
balajis opened this issue Jun 14, 2020 · 66 comments
Open

Submit your bounty entry here #1

balajis opened this issue Jun 14, 2020 · 66 comments

Comments

@balajis
Copy link
Owner

balajis commented Jun 14, 2020

See here for context on the bounty for an open source tool to export your Twitter following and here for notes on what one possible solution might look like.

Please submit your bounty entry by adding a comment below. Keep your entry as a single comment, and include the following:

  1. link to your website or repo
  2. example image or video showing the code working on an account

Feel free to edit your comment later if you want, but please don't add more than one comment as spam.

We plan to review this by end of day on June 21, 2020 at 11:59pm PT but may extend the deadline if there are a lot of submissions or unanticipated complexities.

UPDATE: we'll extend this to June 28, 2020 at 11:59pm PT given all the interest.

Update: June 28, 2020 - Review has begun

Wow! Review has begun.

So many awesome submissions by everyone. It will take us a little while to go through them, but we have begun the review process. Please feel free to reply with any questionsl

@balajis balajis changed the title Placeholder for submissions Submit your entry for an open source tool to export your Twitter followers here Jun 14, 2020
@balajis balajis changed the title Submit your entry for an open source tool to export your Twitter followers here Submit your bounty entry here Jun 14, 2020
@vasa-develop
Copy link

vasa-develop commented Jun 14, 2020

Here is a GIF for a PoC for CLI app:
The current CLI app can filter users through complex queries to select specific users (above 1000 followers, etc.)
I'm working on the Dashboard now & will keep updating this comment as I make more progress.

twitter-dms

Update 1

A Mock Flow for Dashboard UI (Youtube Video)

Update 2

Code is open-sourced here: https://bit.ly/MarketingBird

Under heavy work...I'll add more context about how will the CLI, and hosted version would work in the README soon. If I have time left till the bounty closes, I'll go for an electron app too!
(If this project takes off, we'll shift the stack to Flutter for all-in-one platform coverage)

We don't have a good name for it yet 😅 Feel free to contribute 😉

Update 3

Some results of using the CLI tool. Signups on simpleaswater.com skyrocketed due to the DMs. Best day ever 🥳

members

Update 4

With the current stack, we can fetch 100 followers in about 4-5 secs (running on my laptop). So, doing the math:

Follower count      Est time to fetch
10k followers       400 secs to 500 secs
100k followers      4000 secs to 5000 secs

Update 5

Able to fetch all public data associated with an account and list/rank them using different parameters. Here is how the public data of a single account looks in raw form:

16473957 | Dr. Tayo Oyedeji | @tayooye | Private: 0 | Verified: 0 | Bio: Founder/CEO at @overwoodng (http://overwood.ng ) | Former corporate CEO & university professor  | I help people manage their money. | Location: Nigeria/USA | Url: http://www.overwood.ng | Joined: 26 Sep 2008 1:20 PM | Tweets: 4211 | Following: 392 | Followers: 92384 | Likes: 1614 | Media: 271 | Avatar: https://pbs.twimg.com/profile_images/1091079425624600576/M_ZJ-MN8_400x400.jpg

@brunneis
Copy link

brunneis commented Jun 14, 2020

I coded this Python library. It can extract the followers through the anonymous API. It might be useful...

image

https://github.com/labteral/bluebird

@bconnorwhite
Copy link

bconnorwhite commented Jun 15, 2020

Working prototype built with Node (TypeScript) if anyone would like to work together:
https://github.com/bconnorwhite/twitter-dm

Currently have a basic CLI wizard working. Both follower data and API cursors are saved in a state file incase the script is stopped.

Can add full follower data export (name, location, account creation date, follower count, verified status, etc.), filtering based on these parameters, and a slick webapp, but only if others want to contribute so we end up with one or a few good tools rather than 20 half baked solutions.

dm screenshot

@cmaer
Copy link

cmaer commented Jun 15, 2020

Hi @bconnorwhite and @balajis , I'm also working on Node-based version, along with Express-driven front-end.

Nice work, @vasa-develop !

Will update as I progress.

@mspanish
Copy link

mspanish commented Jun 15, 2020

Update 6/27/2020, 11:45pm Alaska time!

What a couple of weeks! I want to thank @balajis for lighting such a great fire under my rear, as I have done more hard core development and learning in the last 2 weeks than certainly any time after about January 20 of this year. So grazie for that. I had hoped to have my working app online, but as I switched mid stream from Node.js to Python, deploying the my aiohttp app with all of the awesome power of the scraping scripts combined with the official API turned out to be harder than I realized. It will be live within a few days after the Digital Ocean guys get back to me! But for now I'll leave you with my screenshots and my thoughts. This app requires only a Twitter login from the user, with no need for their own API keys.

I spent a lot of time thinking about how you determine which followers to direct message first, given that somebody with 100,000 followers is looking at that taking 3 whole months to do. For people with really big followings, achieving a sorting ranking algorithm becomes vital. This is what lead me to abandon the Node.js app you see in these screenshots! To arrive at a meaningful value for which followers you should DM first, a simple sort by followers is extremely insufficient. In fact, some of the followers with a smaller following may indeed have a much more powerful reach, as seen in the amount of likes and retweets you see in their timelines.

The official free API limits you to just 20 or so tweets, and just 1 week of history. The paid API moves that up to 30 days and more tweets - but that's still just not good enough. People go on vacation, or get busy. So tweets will need to be scraped, and luckily Python has some fantastic tools like Twint that will do that for you. Twint does a much better job at getting tweets than getting your followers, and the official API is actually pretty good at doing that within a few hours even for an account with 200,000 followers. But you need the scraping tools to also collect somewhere between 50 and 100 tweets per user, in order to use them in the ranking algorithm.

I envision a user defined ranking where you can play with the settings a little - giving more or less weight to things like being verified, quantity of followers, ratio of followers to friends, and the average amount of likes and retweets per tweet. You could segment that further if you want to categorize by subject parsed from their content.

You can also check engagement in your own feed. People who have mentioned you will be far more receptive to a DM from you than a follower with 1 million followers who has never mentioned you or given you a RT. The scraped data will be the key for all of these metrics.

I built a rather playful UI, as my background is in education - sorry about that! But here are some of the pages I put together, with a backend built both in Node.js and later in Python (luckily I used Nunjucks templating which transferred perfectly to Jinja2) - with data stored to Redis, although eventually there would need to be both a server side and client side database - I like LevelUp a lot, but don't know if it's the most stable thing to use, so I'd ask around before settling on that. At any time the user can export all of their data to CSV, XSL, or JSON as well.

index

I just picked some random copy for the home page, may not be quite right but I want the app to do MORE than just export your following: I want it to help people grow their social media presence wherever they may want it, via Twitter, their blog, FB, etc.

settings

My idea for this placeholder page is to integrate my own meme maker which generates art for social media using a database I've developed of around 60,000 vector images that can be used to quickly make a cool graphic. It's a huge update to kwippe.com - and the idea here is that the user can craft a really great public page customizing the colors, graphics, fonts, etc. They can also generate images to use in campaigns, as ads may end up being an effective tool, like the ones Bill Gates runs that encourages people to sign up for Gates Notes

dash1

My dashboard is a bit spreadsheety, and that can be made spiffier - but I wanted a quick way to get at the followers (and friends, which will be in a separate tab). This is where the "reach ranking" will be key, and you'll be able to select 1000 more people to DM, and that will be entered into Redis. I also intended to add a shiny STATS section to the top, with things like total followers, total likes and retweets, total reach, etc.

dash2

The page above just shows how you can search throughout the fields. It looks in the description, as well as their screen name and full name, but once you have tweets indexed for the users, you could also choose to include all of their tweets in your search. Twitter offers NO easy way to search just your own followers or friends, so this is a feature I would actually like to use myself just to manage my timeline better.

dash3

When you click on a user's picture - it pulls up their 20 tweets going back 7 days from the official Twitter API. The Python app will try to have much of that data scraped either in realtime, or as a task that keeps going even after you log off. Finding an efficient way to do this without taking up too much memory, bandwidth, etc - is something that would have to be looked at. You could opt to send emails to the user when their data is done filling out, and their metrics are updated.

While I see some beautiful work has been done on the DM campaign, as well as generating affilate links - and those could be integrated here - the other thing I thought about if you wanted to grow your Twitter following as well as export users, is using a hashtag incentive. Here I'm not sure if you could get enough data from the official API, or if you would want to use the scrapers. The app can use both.

Offer users the incentive that if they RT your stuff and include their own username plus a chosen keyword - let's say
#myhandle$$camogli$$ - that for every RT they generate, they earn points on your leaderboard. Then you just have to check your own RTs and see which usernames are found in the hashtags with those keywords. Just a thought!

Great job to all - I've seen some really awesome stuff here on this thread, and I know how hard folks have worked. Good luck to all and I hope you find a good solution to your issue.

@ricardopolo
Copy link

ricardopolo commented Jun 15, 2020

I'm working in this challenge too. I'm very excite to learn a lot 🤩 and create an excellent solution that can win the bounty.

I bet the competence will be hard, especially because @vasa-develop is working in his repo since 2015!!

Good luck for everyone 😄 and thank you to @balajis for making this possible! 🌟

@sandeepvellanki
Copy link

sandeepvellanki commented Jun 15, 2020

For any clarification or demo, send email to vellanki.sandeep@gmail.com.
Will respond to your request as soon as I can.

TwitterDM

The Jupyter notebook that acts as a local application that can send Direct Messages (DMs) to all of your followers based on the keywords in their bio and/or count of followers they have. It allows you to update your followers list with new followers whenever you want to and add these to the list of followers whom you want to send automated DMs to.

This works without any issues for twitter accounts with huge followers.
(tested with an account having ~450k followers, but would work for even bigger accounts as well.)

Prerequisites:

  1. Jupyter notebook

How to:

  1. Download the 'Twitter DM application.ipynb' file to a local folder.
  2. Fire up Jupyter notebook application and open the above file.
  3. Update the Inputs cell. Most of the defaults work.
    Inputs
  4. Run each cell one after other (Cntlr+Enter to run each cell).
  5. 'Install and load ..' cell: installs all prerequsite packages and imports them
  6. 'Backup data' cell: Backs the data as losing it mean running time consuming intial loads.
  7. 'Status Visual' cell: Give a visual representation of followers count, DB status and DMS sent etc. See the screenshot below:
    Sample status chart
  8. 'Update Follower IDS list' cell: Updates any new followers from last run. Intial run will load all follower IDs in Ids table.
    220k records
  9. 'Update metadata' cell: Updates metadata like bio, followers count and so on for the followers.
    meta data
  10. 'Send DMs for today' cell: Filters, Sorts, choose and sends the number of DMs you want to send on a particular day.
    DMsent1

DMsent2

Benefits:

  1. Works across windows and MAC as it is a jupyter notebook. The size of the application is 12kb (very lite application)
  2. Works locally, with flatfiles as backend tables. (can be updated to SQLite, but flatfiles seem just fine)
    1. The Twitter API keys need not be shared with anyone
    2. No worry of your followers data being with a 3rd party
  3. Ability to choose the N followers to send the DMs on any given day based on filters you provide like follower count, keywords in bios, exculded list of followers and so on.
  4. Easy to backup data.
  5. Visually check the following counts:
    1. Followers count at the precise moment
    2. Followers count that have been loaded in local tables
    3. Followers that meet the filter criteria
    4. Followers for whom we sent out DMs (that succeeded and failed)
    5. Followers for whome we can still send the DMS
  6. Update tables with new followers and thier meta data with a click of a button.

First time load times:

  1. With accounts less than 75k followers: ~20-25 mins
  2. With accounts less than 150k followers: 40-45 mins
  3. With accounts less than 225k followers: 60-65 mins
  4. With accounts less then 300k followers: 80-85 mins
    and so on......

Subsequent load/run times are very minimal (under a minute).

Planned Updates:

A Saas tool as follows:

  1. Create a frontend UI that works on twitter login and takes in inputs via a form. Same inputs as above.
  2. Takes to a dashboard that shows the status of your sent DMs and other related metrics.
  3. Ability to schdule DMs.

@daisy1754
Copy link

daisy1754 commented Jun 15, 2020

Here is my submission

Code: https://github.com/daisy1754/tw-megaphone
Hosted version: https://tw-megaphone.herokuapp.com/

It's a rails app that can be easily deployed to heroku or run locally. I'll also provide hosed version but given twitter has per-app DM limit you may just deploy your version on heroku

  1. Twitter login
  2. Sync your follower (5000 followers per 15min)
  3. Tweak ranking strategy via UI
  4. Test send DM to specified follower. Web app also provides a per-user unique link to collect email but you can also embed your own email subscription email
  5. Schedule daily job to send DMs (1000 per day)
  6. Provide export (especially useful if you use built-in link to collect emails)

Screen Shot 2020-06-17 at 9 10 11 PM
Screen Shot 2020-06-17 at 10 01 39 PM
Screen Shot 2020-06-17 at 10 35 37 PM
Screen Shot 2020-06-21 at 12 14 16 PM
Screen Shot 2020-06-23 at 10 39 22 PM

@DeveloperRyan
Copy link

DeveloperRyan commented Jun 15, 2020

These past two weeks have been pretty fun, and @DeveloperHarris and I have definitely learned a lot. Thank you to everyone involved!

Project Pineapple is a collection of tools that we've been working on to address this competition. So far, our CLI tool is completed and our email web app and electron desktop application are still under work.

All the source code is available under an MIT license at https://github.com/DeveloperRyan/project-pineapple.

Project Pineapple CLI - Finished:

2020-06-28_00-34-20

Features:

  • Strong focus on locally stored information, no data stored online giving the user full control.
  • Intuitive and visually appealing CLI menu, allowing anyone to use it
  • Config file that stores Twitter API app keys allowing for a quick startup
  • Local SQLite database creating a seamless setup process
  • Ability to view all total followers and download or "sync" the information locally.
  • Automatically handles API rate limits allowing a person to run once and forget
  • Export all downloaded follower information to CSV, with the option to only pick selected columns if preferred.
  • DM all followers w/ message preview and confirmation
  • DM test user w/ message preview and confirmation
  • Message parsing through handlebars.js allowing {id}, {screen_name}, {followers}, {friends} and other data to be used in messages for personalization
  • Reset config and followers database quickly through dedicated command

Setup:

  • Clone repo
  • With Node installed, run npm i in the project-pineapple\cli folder
  • When complete run node ./index.js

Pineapple Project - Electron Local App - Under Construction:

Planned Features:

  • Easily visualize followers
  • Create dedicated campaigns targeting specific groups of users with customized messages
  • Comprehensive dashboard providing quick-glance overview of profile and campaign stats
  • Many, if not all the features of the CLI

Auth
Dashboard
Follower DB
Campaigns
Campaigns-modal


Pineapple Project - Email List /
Referral Webapp - Early Stage Construction

Planned Features:

  • Login with Twitter
  • Create an email list w/ dedicated referral link to use
  • When a user clicks email list referral link, ask them to sign in and ask explicit consent to join the email list
  • Give signed up user, personal referral link to share with other potentially interested users
  • When follower refreshes email list page they can either remove themself from the page or see the current scoreboard

We plan on continuing these projects (after a short break) to increase our portfolio and add to our resumes. Please let us know any feedback you have, and any features you would like to see. If we receive enough interest, we might spin this out into a more fleshed out open-source project.

@IvanMontillaM
Copy link

IvanMontillaM commented Jun 15, 2020

2020-06-15T18:13:16Z: Working on this after my work time.


TwExodus

(Link to repo)

My project takes care of the stealthiness required to do this follower base exodus. A prototype mostly functional is at the src/unstable directory.

Not only it does send Mass DMs, it can also extract email addresses from DMs and dump these in a CSV file for easy importing.

If you require help at testing it, I am available at Twitter/Facebook/Instagram/LinkedIn using the same username I'm using here at GitHub.

@transitive-bullshit
Copy link

transitive-bullshit commented Jun 15, 2020

Here's a pretty robust Node.js solution: twitter-flock.

It uses Twitter OAuth and stores the state of each BatchJob in a serializable, resumable format in order to be as robust as possible.

It also includes a simple Workflow abstraction for aggregating sequential BatchJob tasks in the same serializable and resumable format.

I'll be using Saasify to turn this core functionality into a full, hosted SaaS product. You can think of Saasify as a Shopify for SaaS that handles all of the tedious OAuth, billing, and boilerplate for SaaS products like this.

Some basic stats:

  • Exports the user ids for all of your followers (~75k every 15min)
  • Resolves those user ids to full user objects (~90k every 15min)
  • Sorts users based on a basic metric of popularity
  • Sends templated DMS (1k per day)
  • Supports exporting state to disk or db and pausing / resuming processing

I don't think the DM approach will be the best solution as discussed in #3 and #4, so I'm currently working on an alternative approach that tries to infer emails for Twitter users (like an MVP of clearbit which is a very difficult sub-problem in and of itself).

Twitter Flock
Screenshot of an example DM (handlebars template is used for the message body)

I'll be updating this comment with progress over the next few days.

Happy to collaborate on the OSS repo if anyone's interested. 🔥 😄 🔥

@treylorswift
Copy link

treylorswift commented Jun 16, 2020

Flybird Desktop App

The Flybird desktop app runs on Mac and Windows and automates sending of direct messages to your Twitter followers, with a focus on soliciting newsletter subscriptions. Keep your API keys on your own machine, send up to 1000 messages per day.

GitHub repo at https://github.com/treylorswift/FlybirdDesktopApp

Flybird Web App

The Flybird web app provides users with a quick and simple way to create a newsletter sign up page at https://flybirdy.herokuapp.com. It tracks all your subscriptions as well as the referrer responsible for generating the sign up. The "Referrals" tab shows you who has brought the most subscribers to you.

GitHub repo at https://github.com/treylorswift/Flybird

Desktop App Features

  • Cross platform application built using Electron & Node.js

  • Downloads your complete follower list from Twitter at roughly 300k followers per hour. Download can be resumed later if interrupted. Followers can be updated/re-downloaded again later.

  • Followers are displayed in a table and can be sorted by "Most Followers" (your most influential followers) or "Most Recently Followed" (the people who followed you most recently). Filter results by entering tags which will be matched against each user's Twitter bio.

  • Message composition has a primitive template engine which allows the user to insert the follower's Twitter handle into the message wherever they want. The intended use case is to insert the follower's Twitter handle in the referral id of a newsletter sign up link.

  • Message sending history is tracked to ensure each follower is contacted only once. Message sends are scheduled to avoid hitting Twitter API rate limit errors.

  • A separate "Sandbox" message history allows you to simulate sending of messages to see how the program operates without sending real messages.

Integration with Flybird web app

The default outgoing message directs followers to the user's subscription page with a unique referral code for the follower. For example if you are 'treylorswift', the default outgoing message will be:

Hey there, are you interested in receiving my newsletter?

You can sign up at https://flybirdy.herokuapp.com/subscribe/treylorswift?twRef=${followerTwitterHandle}

When messages are sent, the follower's Twitter handle will be included in the link. They can share this link with their audience and they will receive credit for anyone who uses the link to sign up.

For example if I send the above to @balajis, he will receive the link https://flybirdy.herokuapp.com/subscribe/treylorswift?twRef=balajis

If he shares that link with others, and others use that link to sign up, he will get credit on the referrals leaderboard displayed on the Flybird web app.

Pre-Built Desktop App Binaries

Developer Install

Further work being considered:

  • A "hosted site" version of the above Electron app where people can login with their Twitter account and get right to work.

  • Expand on the template variables that can be used in the outgoing DM's to customize each message further.

  • Allow users of ITK Sign Up to provide an API key to kickbox which can score email addresses submitted to the sign up engine

Command Line Tools

My command line follower downloading, caching, and message sending engine (the same engine underlying the desktop app) is currently located here:

https://github.com/treylorswift/InfluencerToolkitCLI

updated June 24th with new desktop app, newsletter sign up web app, and improved back-end / command-line engine

updated June 26th with the ability to send unique sign up links (with referral id's) to every follower

updated June 28th with an improved subscriptions management UI and new name and logo

@OlabodeAbesin
Copy link

OlabodeAbesin commented Jun 16, 2020

Just seeing this, will be back in few days

@balajis
Copy link
Owner Author

balajis commented Jun 18, 2020

Nice work everyone. Am looking at these over the next few days along with other submissions as they come in.

Please also take at this writeup on a complementary affiliate link approach:
https://github.com/balajis/twitter-export#the-affiliate-link-approach

As well as this proposed scoring mechanism:
https://github.com/balajis/twitter-export#bounty-scoring

It's possible to combine the mass DM and affiliate link approach by sending a mass DM to a selected subset of people, each of whom gets a custom signup and affiliate link. @thumpri put together an excellent infographic on this:
#9

To answer a recurring question, any email list we get can be exported/imported to any site given the consent of the users (here's how that works for Substack.

One thing we're thinking about is how to give folks partial credit. One possible approach is that the $10k bounty goes to the final tool, and the remaining funds are split to give individual prizes to folks whose code or writeup helped contribute to the answer. The overall goal is that everyone learns something and has fun by participating, of course.

@shea256
Copy link

shea256 commented Jun 18, 2020

Hey everyone, love all the submissions in here!

Gave this challenge a try with a web app that:

  1. lets you provide database credentials and twitter credentials
  2. logs your followers to that database
  3. lets you message them

Code

https://github.com/shea256/movement-for-twitter

Live App

NOTE THE APP WILL NOT WORK UNTIL YOU SET THE DATABASE AND TWITTER CREDENTIALS ON THE SETTINGS PAGE - THIS WILL BE SIMPLER IN A FUTURE VERSION

https://movement.vercel.app/

Screenshots

Dashboard:

Screen Shot 2020-06-18 at 11 08 32 AM

Followers:

Screen Shot 2020-06-18 at 12 27 33 PM

Settings:

Screen Shot 2020-06-18 at 12 31 45 PM

Future Work

Logging

The logging experience is a bit janky for now, so I recommend carefully reading the instructions in the README.md. Even better, I recommend running the app locally and watching your console logs so you know when the logging has completed. This limitation is just temporary. I plan on spending another day or so on it to create a queue that will log all of the followers in a more reliable and convenient way.

Messaging

The message feature currently only lets you message one person at a time and it doesn't show you a history of the people you've already messaged. A future version will have support for this.

@JordanDworaczyk
Copy link

JordanDworaczyk commented Jun 20, 2020

An implementation using Python: https://github.com/JordanDworaczyk/Twitterbot

help_connect

Summary

Twitterbot consists of a command line interface for downloading followers, ranking followers, and
then sending out a message to each follower based on their priority which is determined by their
ranking.

Using Twitterbot a user can download up to an estimated 75,000 followers every 15 minutes and send up to 1000 direct messages each day.

@MohitKumar1991
Copy link

MohitKumar1991 commented Jun 21, 2020

Hey everyone, great submissions!

Here is my webapp with the salient points:-

  1. Can survive bad internet and run for any number of days
  2. Keeps all followers cached locally (easily works for 100k followers)
  3. Has a webapp to explore dataset as well as see status
  4. Support sending DMs

Code

https://github.com/MohitKumar1991/twitter-export

ezgif-6-bde1c9f707b4

Planned for Next Week:-

  1. Fix bugs and improve overall stability
  2. Improve UI
  3. Message History & export followers

@CodeHownd
Copy link

CodeHownd commented Jun 22, 2020

Hey guys,

I created a python CLI and GUI app for the DM Link Approach

Please feel free to contribute or clone the repo to add improvements

I am open to collaborating for a web app for the DM Link Approach or a new solution for the Affiliate Link approach.

I am attaching the code below
https://github.com/CodeHownd/twitter-export-followers

image

GUI demo
https://streamable.com/5t7oqd

@petabite
Copy link

petabite commented Jun 22, 2020

Twitter Blast

Sup guys, these submissions looking 🔥 so far!

Here's my solution to the mass DM approach: https://github.com/drizzleco/twitter-blast

Here's the hosted version: https://twitter-blast.herokuapp.com/ (👏 to @fangherk)

My solution uses Python, tweepy, Flask, and SQLAlchemy. Check out the README in the repo for more in-depth documentation. Questions and suggestions are welcome!

CLI Version

twitter blast

Flask Version

twitter blast
the dropdown menus didn't show up on the recording for some reason :(

Features

  • easy authentication using Sign in with Twitter
  • preview follower rankings before sending the real deal
  • defaults to dry run mode to prevent unintentionally sending out DMs
  • remembers when a DM has been sent to a follower so no unintentional double sends
  • automatically pauses execution to wait out rate limits

Getting Started(CLI version)

  1. make install to install dependencies

  2. Edit secrets.py(automatically created) in the same directory as twitter_blast.py and add your app credentials:

    • make sure your Twitter app has "Read, write, and Direct Messages" permission
     HOSTED_CONSUMER_KEY = "" # for the flask app
     HOSTED_CONSUMER_SECRET = "" # for the flask app
     CONSUMER_KEY = "" # for the CLI version
     CONSUMER_SECRET = "" # for the CLI version
     SECRET_KEY = ""
  3. On first run, you'll be prompted to authorize with Twitter

     $ python twitter_blast.py
     Visit to authorize with twitter:  https://api.twitter.com/oauth/authorize?oauth_token=_______________________
     Paste the verification code here: ________
    
  4. python twitter_blast.py fetch to fetch your followers

  5. python twitter_blast.py preview to test out the ranking system and see how your followers will be prioritized

  6. python twitter_blast.py send to dry send a DM to your followers(add --real to send them for real!)

Getting Started(Flask version)

  1. Complete steps 1 and 2 from above.

  2. Add http://127.0.0.1:5000 to your callback URLs in Twitter dev app settings

  3. make start

    OR

  1. source .venv/bin/activate && python app.py

Behind the Scenes

  • Fetching followers data
    • fetches ids of followers first using followers/ids
      • followers/ids returns 5,000 user ids/request(max 15 requests every 15 minutes )
      • TOTAL: 75,000 users every 15 minutes
    • then, fetches user object using users/lookup
      • users/lookup can get 100 user objects per request
      • with user-level auth, you can make 900 requests every 15 minutes
      • TOTAL: 90,000 users every 15 minutes
  • Ranking Followers
    • uses SQLAlchemy database queries to do the heavy lifting
  • Sending DMs
    • uses tweepy's wrapper for direct_messages/events/new (message_create)
    • updates database to keep track of which followers have been sent DMs

@moesalih
Copy link

moesalih commented Jun 25, 2020

I created a web app based on the referral link approach: https://socialexporter.web.app/

Code: https://github.com/moesalih/socialexporter

socialexporter-home

Once an influencer logs in with their Twitter account. A public page gets created to ask their followers for their emails. And the home page shows the link to that page and some other links to export all follower emails to a text file, see the referral leaderboard, and change settings.

socialexporter-loggedin

When a follower opens that public page, they get a simple email form to sign up, and once they do, they see a custom referral link to the same page they accessed and get assigned a follower ID.

Here's my public sign up page: https://socialexporter.web.app/moesalih_

socialexporter-userpage

socialexporter-userpage-after-signup

The follower can share this custom referral link with others, and get credit for anyone who signs up from that link.

A public leaderboard page will show the top follower IDs and how many others they referred. If this leaderboard is accessed by a follower who submitted their email and referred others, then their follower ID gets highlighted (like bellow). If this leaderboard is accessed by the influencer, then they see the emails of the top followers (in case they want to contact them).

socialexporter-leaderboard

I'd love to hear feedback on what I have so far, and if you have any questions.

This still doesn't include determining the quality of the emails, preventing referral abuse, and the payments part. And there's lots of improvements that can be done, but this is a functional MVP of the approach.

@kanav99
Copy link

kanav99 commented Jun 26, 2020

Hi there!

We (me, @supra08 and @palakg11) have made the submission application at these links -

  1. MacOS App - https://github.com/kanav99/TwitterCampaigns
  2. CLI - https://github.com/supra08/twitter-campaign-cli/
  3. WebApp - http://ec2-54-161-90-135.compute-1.amazonaws.com:8080/

TC0
TC1

See our application in action -


Command Line App -

Web App -

Our app provides an elegant way of reaching the twitter followers with the mass DM approach. This MacOS app has the benefits of native performance giving it an edge over web versions. But for non-mac users we have also prepared solutions. Those tech savvy guys who want to get their hands dirty can use the CLI with loads of features and campaign deployments plans that gives you a quick way to reach your followers.
For those who want a quick web solution, we have a react based web version deployed
The deployment plans give an extimate duration and you can plan you DMs accordingly. The processes run in background and be paused and resumed at ease.
We can modify campaigns with follower specific plans and also verify whether a particular follower has received a DM or not.

@snowdot
Copy link

snowdot commented Jun 27, 2020

Hello everyone!

https://outfluencer.co/ is my solution for this challenge! It is a hosted application coded in JavaScript that runs on Node.js and uses MongoDB as a database.

Requirements to use the app

The app requires minimum access permissions (read only) just for the Sign in process. Additionally, to use the tools it provides, you need the following keys (session storage only):

  • Twitter API keys with access permission set to 'Read, write and Direct Messages';
  • Coinbase API key with wallet:transactions:send permission (for enhanced security it is recommended to whitelist https://outfluencer.co/ 's IP address) - note that the rewarding system is scheduled for future development;

Outfluencer.co provides Twitter influencers with the following tools (current working version):

  • Download followers and/or subscribers (people that shared their email) in .csv files ✓
  • Send automated mass direct messages ✓
  • Sort and filter data before sending DM (e.g popularity, activity, verified accounts etc) ✓
  • DM with quick replies options ✓
  • A public page https://outfluencer.co/subscribe/username where anyone can agree to subscribe and share their email ✓
  • Referral URL https://outfluencer.co/subscribe/username?aff=username to expand the affiliate network ✓
  • A leaderboard public page https://outfluencer.co/leaderboard/username with ranked subscribers ✓

gif

Sign in

You log in with Twitter OAuth, enter your API Keys (session storage) and start extracting your followers. On average, it fetches approximately 3.000 followers/sec for those accounts with less than 75.000 followers, after that, the 15min Twitter API restriction kicks in (max. 75.000 followers/15 min).
note: entering your API Keys is mandatory until you extract your follower DB (you'll get an error if your keys are incorrect), after that, you can leave the inputs empty BUT only to see the dashboard (if you want to send DM your keys must be correct).

keys

Dashboard

After the followers have been extracted, you get redirected to the Dashboard where you can download your data as .csv files and have an overview of the numbers behind it.

dashboard

Mass DM

This is where you can sort and filter you followers and start sending mass DMs. I've added some little things to help you write more personalized things, such as writing {{follower}} to insert the follower's name or {{url}} to insert your public page or quick replies in case you expect an answer from the recipient.

massdm

Campaigns

This page acts like history, keeping records of the mass direct messages you've sent, showing some 'must have' details.

campaigns

Collecting valid emails with 3 clicks:

  1. click: the follower gives his explicit consent;
  2. click: the follower initializes the Twitter app;
  3. click: the follower authorizes the Twitter app to share his email address;

Creating this app with the sole purpose of collecting emails was the best approach considering the following:

  • it targets only Twitter users;
  • there is no need for email validation;
  • it eliminates fake signups;
  • in my opinion, it is also compliant with Twitter's Rules and policies, section C. Automated Direct Messages, paragraph Interacting with users via Direct Message that states the following:

If you will be asking a user to provide personal or private information via an automated Direct Message, you must clearly explain how you will use the information you’re collecting. Consider including a link to your privacy policy in your Direct Message to the user, as well as in your Twitter profile bio.

subscribe

authorize
note: the follower can subscribe to multiple influencers and the app will ask for authorization each time.

Leaderboard

On this page all the subscribers will be displayed and they are ranked by the no. of people that subscribed as well.

leaderboard

Affiliate

After someone subscribes, they are redirected to this page where an affiliate link is generated.

affiliate

Payment proposals for the rewarding system

1) Automated international payouts by integrating Coinbase's API

From a dedicated page, similar to Mass DM, the influencer will be able to create a New Reward Campaign by setting:

  1. Parameters for the campaign:
  • the duration for the campaign to run;
  • the total amount of bitcoin for the reward;
  • the maximum amount of bitcoin per individual;
  1. Criteria to determine the winners:
  • the number of people to be rewarded (top 3/ top 10 etc);
  1. Write the message for a note that will be included in the email that the recipient receives (from Coinbase); This feature would only be available for higher amounts.

In order to activate the Reward Campaign, the influencer will have authorize it by entering his/hers Coinbase API Keys.

Under the hood, the following steps will be taken:

  • for each campaign created a new bitcoin address will be generated for his/hers primary Coinbase wallet;
  • the total amount of bitcoin will be sent/moved to the newly generated address (it is his/hers own address and the funds come back to the primary wallet);
  • when the reward campaign ends it will start to streamline payouts: sending the maximum amount of bitcoin to the winner's email address;

The pros and cons of integrating Coinbase

Pros:

  • micropayments between Coinbase accounts are free and fast because they are off-blockchain transactions (settled on their internal ledger);
  • it gives the winner the possibility to exchange his/hers winnings to another preferred currency;
  • it only requires the email address (which it was already collected) to make the transaction, eliminating extra steps to collect or generate additional sensitive data (e.g. bitcoin address);

Cons:

  • the winners are required to have a Coinbase account;
    workaround: since the web app already has a built-in system to send direct messages, for those transactions resulting in an unverified_email error, you could opt to send a direct message informing the subscriber that he/she won and needs to register at Coinbase to claim the reward;

2) Payments by integrating with Tippin.me (not confirmed yet)

fast, international, automated, small payments to users for decentralized work

... it made me think about micropayments with Bitcoin using Lightning Network, more precisely it made me think of the Tippin.me app. I do have limited knowledge in this area, but I'm waiting for a response from Tippin.me to see if it is feasible to build an automated tipping process for the rewarding system and also if they are open for collaboration in this regard.

Future developments

  • Implement one of the above proposed Payment proposals for the rewarding system;
  • Build an opt-out of receiving automated direct messages solution and create a blacklist with followers that did, thus making sending mass direct messages more compliant with the Twitter Rules and policies;
  • Add additional filters to target specific followers (e.g. collected emails, hashtags etc);
  • Admin functionality to allow the influencer to customize his public page;
  • 'Stop button' to stop the automated direct messages from sending;
  • Set up welcome message that will be sent automatically to new followers;
  • A page with data visualizations;

Github repository

https://github.com/snowdot/outfluencer

Final thoughts

This was a fun and interesting challenge and I'm glad to be part of it. I believe that with this project we've only scratched the surface and that there is an immense potential to build great things with the Twitter API.

@thumpri
Copy link

thumpri commented Jun 27, 2020

Really amazing submissions everyone. I spend a lot of time empathising with the problem and figured out the solution approach as explained through the following mind map and mentioned in issue #9:
Follower Export Bounty

While working on the Hybrid approach, I encountered another problem and further modified the solution to the following : 




Hybrid Approach


As explained in the above mind-map, I have built a ranking algorithm which serves the purpose and the code for which can be found here : https://github.com/thumpri/Soporter

While coding the proposed solution, majority of my time was spent analysing the Twitter API and finding loopholes to surpass the limits.
 Following are my findings :

  1. Twitter API Limits Loophole - 5x return speed
  2. Removing the app limits
  3. Get Followers - fastest non-api based method
  4. Get tweets - fastest non-api based method

.
    These are better explained in the following mind-map.

Findings

I think we’ll have a brilliant tool if we keep this algorithm as the base and add functionalities like-
• The front end code by @snowdot or @moesalih
• Maintaining an apt database which preserves the state as done by @MohitKumar1991
• generating Auth bearer token for better speed (5x)
• Twitter authentication, sending DMs as done by in Python by @petabite and so many others.

I would love to continue working on it as a side project and bring all the pieces together. This has surely been a great learning experience for me to encounter a seemingly easy problem, iterating through several solutions and finding the one that fits (for now!). :D 

Thank you @balajis for hosting this. I hope I was able to add value to it.

@syllogismos
Copy link

syllogismos commented Jun 27, 2020

Eschernode!!!

Messaging Automation for Social Media Influencers XD

https://eschernode.com/

Video Link: https://www.youtube.com/watch?v=4-OiEdxJxKw

Basic overview of the UI. You sign up, update keys, start indexing followers.

Filters
And then go to the Filters page. Create whatever targeted segments you think are relevant. Currently the filters I provide are, follower count, friend count, keywords, muted by you, verified, last active, twitter handle to test your messaging campaigns. I have more filters in pipeline.

Campaigns
After you create your targeted segment, you start your messaging campaign. There you can enable a custom link or subscription link I provide to send with the message. Using these links I track, clicks and conversions. conversion is when the user successfully subscribed. I built the tracking of conversions and clicks. /subscribe/user link. These things will be shown on a campaign page. I will also be tracking other metrics for the next day after the campaign is sent.

I currently disabled the actual campaign sending part. Its actually scary, its too powerful, worried I might accidentlly send it to all my followers becuse of a bug or etc. I want to test more and then enable it. Also it feels like I over engineered this a little bit in two weeks. I used workers and message passing. Lots of things can go wrong. You can send a test dm tho for now. It's to test right before you start a messaging campaign.

Even though you wont be able to start campaigns, You can sign in and index your followers, and then play with the segmenting your followers. Its really fun. The filters UI came out really well.

If people start using this, I will be implementing A/B testing of your campaign messages, tracking unfollows right after campaigning, more checks to send dms based on whether a follower received a message recently.

Please sign up and let me know if there are any bugs. I hope not!!

Thanks @balajis for hosting this fun product challenge.

I'm really happy with the Filters UX, you add an extra filter by clicking on the plus, and you can delete a filter by clicking on the < next to the filter.

img
img
Imgur
Imgur
Imgur
Imgur

Update:

link to the cmd line tool I built initially that supports the backend to this webapp https://github.com/syllogismos/balaji

Implementation Details and Technologies Used:

  • React Frontend
  • Django for API
  • gunicorn webserver
  • dramatiq for asynchronous producer consumer implementation
  • rabbitmq for storing messages of dramatiiq
  • elasticsearch for searching twitter users
  • firebase for user auth and hosting the webapp
  • tweepy

@socionity
Copy link

socionity commented Jun 27, 2020

JMEL - JoinMyEmailList (jmel.org)

A BTC Affiliate + Mass DM approach

A simple tool to let influencers create a link using which followers can signup to their newsletter and get rewarded in BTC
Repo : socionity/jmel

Influencer flow

  1. Influencer signs up with Twitter account, granting full permission to account
    home

  2. A link gets created, a bitcoin address is generated

  3. Influencer fuels the bitcoin address
    btc

  4. Influencer types a message that is to be displayed before the users sign up, from the settings tab
    settings1

  5. Influencer goes to the Invite tab and invites the top 1000 followers by sending them a DM (can be repeated multiple times, over days)
    invite

  6. Influencer can track who all have signed up on the Subscribers tab
    subs

Subscriber Flow

  1. Subscriber receives a link
    join1

  2. Subscriber logsin with a social media account giving permission to access email

  3. Once registered, an affiliate link and a scratch card is generated

  4. Scratch card can be opened once in 24 hours

  5. More the number of people who have registered using the user's link, more the likelihood of winning on the scratch card

  6. Subscriber gives the BTC address and opens the scratch card
    Screenshot from 2020-06-28 01-08-32

  7. BTC is transferred to the given address. Next scratch card will be available in 24 hours. All affiliate sign ups from this point on will be considered for the next scratchcard's probability.

Self Hosting

  1. Clone this repo
  2. install MongoDB
  3. Create Twitter Apps - one with read write & email permsission, another with read and email permission
  4. Edit config.js
  5. npm start

@npgeorge
Copy link

npgeorge commented Jun 27, 2020

Introducing Rabble, a Flask App developed in Python to send prioritized mass messages to your Twitter followers.

The repo can be found here Github Repo for Rabble: A Twitter App for Mass Messaging your Followers

Rabble: A Twitter Application for Mass Messaging your Followers

Login with your Username

login

Mass Message your Followers

message_proof

Prioritize Mass Messaging using Pre-Sorted Groups

send_message

Sort By Location to Target Specific Followers

sort_by_location

Export your Followers to a CSV File

export_new_york

Link to App

Rabble: A Twitter Application for Mass Messaging your Followers

How it Works

Use this app to gather, sort, and send a message to a database of your Twitter followers.
Sort your follower database with 8 unique parameters. Export your followers to a CSV file.

Mass Messaging

Choose how many followers you'd like to reach. Choose a parameter to prioritize them by.
Filter by location if you so choose. Write a message and inform your twitterverse followers.
Twitter only allows 1000 DM's to be sent per day. With large amounts of followers, this can take a while.
Feel free to set it and forget it. This app was designed on a timer with Twitter Rate Limits built in.

Export to CSV

Export your followers from a pre-sorted database to a CSV file.
Load time is approximately 90 seconds per 5000 followers, patience is key!

User Guidelines

Users need to ensure they have "Read, write, and Direct Messages" enabled on their app tokens.
Twitter Apps

twitter_permissions

Note

Due to heroku timeout limits at 30 seconds, the GIFs above were locally hosted.

@ShreyasJothish
Copy link

ShreyasJothish commented Jun 28, 2020

Twitter Exporter

Twitter Exporter is designed to be a desktop application with single user in mind. However this solution can be scaled
to cloud based solution with added security features like OAuth sign-in and higher performance database with minimal
code changes.

Key Features

  1. Security keys and follower data stored on locally on user's system and not shared outside.
  2. Visualisation of fetch subset of follower's information from twitter for analysis before sending DMs.
  3. Multiple filter options to select followers to send DM with subscription links.
    Currently available filters:
    a) Filter followers based on their twitter joining date.
    b) Filter followers based on minimum number of their followers.
    c) Filter followers based on minimum number of their friends.
    d) Filter and categorise high value followers (power users) based on maximum number of their followers.
    e) Filter and categorise high value followers (power users) based on maximum number of their friends.
    f) Filter followers who are verified by twitter.
  4. Provision to send DM and retry DM after configured number of days. Follower name will be auto filled in DM.
  5. Test functionality to try DM and retry DM on up to 5 configured twitter accounts.
  6. Visualise statistics on screen.
  7. Hover over individual marker on scatter plot to get details about specific follower like number of name, follower count, friends count and DM status.
  8. Option to export data as csv.
    a) High value follower information currently fetched.
    High value follower is identified based on their respective follower and friends count.
    b) Follower information currently fetched.
    c) DM status with timestamp mapped based on follower id.
    d) Skipped follower (followers for whom you do not have access to send DM) information currently fetched.
  9. Automated fetch and DM option based on twitter rate limit.
  10. Auto refresh of visualisation and summary.

Git Repo

https://github.com/ShreyasJothish/twitter-export

Configuration and Setup details.

https://github.com/ShreyasJothish/twitter-export/blob/master/README.md

In Action

Twitter Exporter Dash Board with Statistics

Screenshot from 2020-06-28 07-42-12

Individual Follower Overview

Screenshot from 2020-06-28 07-42-35

Data Selection

Screenshot from 2020-06-28 07-42-59

Zoom In

Screenshot from 2020-06-28 08-02-15

Export

Screenshot from 2020-06-28 07-44-04

Future enhancements possible

  1. Support multiple users by integrating with existing web application or new deployment.
  2. Support OAuth based authentication.
  3. Make use of better performance database like PostgreSQL or MongoDB.
  4. Support dynamic configuration updates.
  5. Additional filters based on twitter user information.
  6. Special configurations for high value followers.
  7. Support affiliate link solution.
  8. Word Tokenization and other NLP based filter on follower's description.
  9. Additional visualisations.
    and much more.

Twitter Exporter is developed on Flask framework and many of enhancement can be achieved with minimal code changes.

@ghost
Copy link

ghost commented Jun 29, 2020

Social Exporter

Hi, we are a group from Australia and we have created social exporter (socexporter.com). We implemented both the direct DMs and affiliate campaign solutions. An influencer can choose to run both if they choose.

Design

Social exporter relies on 3 components. There is a front end web app implemented in Flask, a multithreaded accounting software program written in c++, and bitcoind to create payments and accept deposits.

The flask front end uses sqlite to store information about users and influencer data and redis to for queues and job processing. It uses rq to allow for follower importing and the sending of Dms to be done in worker threads so the website doesn't become unresponsive with multiple users.

The accounting software uses mysql and implements user balance tracking and management in a similar way to a crypto exchange. When an influencer or an affiliate requests a payout the flask app sends the requests to the accounting software for validation before the accounting software notifies bitcoind to make the payment via RPC.

We utilise bitcoind's ZMQ interface to notify the accounting software of a deposit into a users address.

Note about bitcoind

We are using BCH not BTC. The mempool backlog and high fees on BTC make it unsuitable for this project. We thought it would affect user deposits and payouts too much and degrade our overall user experience. Switching to lightning or liquid would be possible in the future when the developers of those applications say that are stable and production ready, as far as i know right now they are not.

Direct Dms

When an influencer chooses to start a direct DM campaign they need to authorise our app to their twitter account. After authorisation we import their followers to our site. Once importing is complete we automatically start to send our DMs to the followers at a maximum rate of 1000 per 24 hours until all followers have received a DM. Due to the daily DM limit, we did not implement a robust ranking system to determine who to send DMs to first. We came to the conclusion that if you have so many followers you need to rank them to determine who should get a DM first, then you are much better off using the affiliate campaign.

Affiliate campaigns

An influencer can choose to run an affiliate campaign. They choose how much to payout a day and how many days they want to run the campaign. If they have enough balance on the site the campaign will begin. The top 3 affiliates will get paid a proportional amount to how many people they have on-boarded that day. Our flask app uses a cron scheduler to run payouts every 24 hours.

Other Features

  • An influencer can export twitter follower data and pepople who have subscribed to your campaign as csv
  • Users who are not influencers can sign up for multiple influencer campaigns and spread links to join those campaigns. Each campaign gets its own referral link.

Future Plans

  • We built our platform to easily accommodate multiple social media platforms in the future. Adding something like Instagram is trivial. We plan to expand the number of platforms in the future.
  • We currently have a flask front end but have found it a little limiting. However, due to time constraints we continued with it anyway. In the future we plan to migrate the flask portion into a vuejs application with a better UI.
  • More testing and bugfixes. Everything works but we consider what we made an MVP if we were to launch it as a product. There are probably a few bugs in there too.

Note about our development environment

For development purposes we have been running bitcoind on the regtest network to avoid loss of real funds. There was an small snippet about scoring but we aren't sure how scoring will be done. Simply due to running on regtest right now some of the functionality (such as making a deposit into a user account to start a campaign) is only available locally. We would be more than happy to either be present during some testing so we can fund accounts with regtest coins or change the network to run on testnet instead (we cannot provide testnet coins). Please let us known on this matter. We are located in Australia.

Screenshots

these screenshots can also be found in the repo here: https://github.com/socexporter/social_exporter_public

socexport_1

to give the user a balance we generated coins on regtest and send them to the address of the user using the sendtoaddress rpc command

socexport_2
socexport_3
socexport_4
socexport_5
socexport_6
socexport_7
socexport_8
socexport_9
socexport_10
socexport_11
socexport_12
socexport_13
socexport_14
socexport_15

@femyeda
Copy link

femyeda commented Jun 29, 2020

bt | Just the beginning

By The Individual Company

bt download
bt demo
repo

What does "exporting twitter followers" mean?

An influencer has an account on a platform separate from Twitter and wants his or her followers on Twitter to migrate and use that platform. The ideal situation is having every person following the influencer on Twitter to migrate to another platform where that influencer is available. However, there is no simple way of doing this. Furthermore, different methods have both PROS and CONS.

Approach 1: Mass DMs

Extract all followers currently following an influencer and then send out 1000 DMs daily such that every follower is eventually notified of the new platform.

PROS:

  • Every follower is notified of the new platform

CONS:

  • Followers may get annoyed, or not remember the influencer and think it's spam and unfollow.
  • If someone unfollows during the mass DM campaign prior to receiving a message they may complain impacting influencer brand.
  • If every influencer starts using an application like this, inboxes will start filling up and people will get annoyed

Approach 2: Affiliate Link Approach

Affiliate link based competition for followers.

PROS:

  • Growth is organic and natural
  • Active followers can be rewarded financially for attracting migrations

CONS:

  • Cannot directly control behavior of those with affiliate links, some may be too pushy and the influencer may be seen as spammy or unprofessional. Moderation is required to ensure those sending out the affiliate links follow a code of conduct.

Approach #BT: Engaged Export

Rather than simply DM all followers of an influencer, DM followers who are most actively engaged with the influencer's content. A engaged follower is one who is retweeting or mentioning an influencer. They are not passively following an influencer, but actively consuming content and redistributing it or engaging in it. In a sense, they are an influencer's most active fans.

We hold these assumptions on engaged followers:

  • A follower who retweeted or mentioned an influencer is more likely to migrate to a new platform if an influencer sends them a DM compared to a follower who has not.
  • A follower is most likely to migrate if a DM is sent immediately after the follower has engaged with the influencer and the probability of migration decreases as more time passes.
    • I.E. it's better to send a DM on the same day of engagement than 100 days after engagement.

PROS:

  • DM only followers who engage with your content, low risk of being seen as a "spammer (followers still receive at most one DM)
  • Can be run daily or throughout the day as you are tweeting, the "engagement" can be calculated at any time and update the "rankings" of engaged followers.

CONS:

  • Not all followers are engaged, so they won't get a DM (But perhaps it's best to wait until an inactive follower becomes active, even if it's weeks down the line?)

How does BT work?

BT is an Electron App. We'll be opening the code base up pretty so anyone who clones the Github repository can build it themselves to ensure there is no malicious code being executed. After all, this is a bitcoin bounty, We have a hunch those sending Bitcoin prefer to keep their Twitter API keys local.

https://www.electronjs.org/

Furthermore, we use json files as our "database" (npm package conf). It is not recommended to close the app when it is processing a "campaign" or there will be data lost on who has been sent a DM in that campaign.

Use Case

Influencers load this app up to periodically build engagement and run a new campaign so that DMs are sent to their most engaged followers. These followers then, due to being engaged, are more likely to sign up to a newsletter or register to a new platform.

Signing In and Navigation

To use the application influencers must enter their Twitter API keys and their Twitter Handle. Once signed in they can navigate by clicking the title of the page they're on.

Build Engagement

On the "Followers" page influencers will be able to see a button called "Build Engagement". Once clicked, this will scan their current timeline for up to (800*) of their tweets and log their statuses in a "tweets.json" file. A set of (25*) tweets that had the most retweets since the last retweeter scan for them will have their last 100 retweeters pulled and logged into the "users.json" file. The last (800*) mentions are scanned and logged into the "users.json" file. Each "user" in the users.json has all their retweet and mentions logged when found in a scan. Finally, a ranking score is calculated and then the most engaged users are sorted on this score and saved to "rankings.json". This scan builds on itself, data is not removed except for new rankings which are generated after every scan. By clicking the "export" button they can save a csv file of their most engaged users.

*This number is variable, but not yet changeable in the app GUI.

A "retweeter" look-up can only scan the last 100 retweeters of any influencer's tweet. Influencers that have tweets with more than 100 retweets may not get a complete rankings analysis. However, periodically scanning throughout the day as more people retweet a tweet will enhance the analysis.

The current ranking formula is just: retweets + mentions by any specific user
We think the formula can be enhanced by adding a time component, but we would also like influencer input on what type of formula could best calculate who is the most "engaged" .

Once the scan is done, influencers will see the rankings of their most engaged users.

Campaigns

On the "Campaigns" page influencers are able to write up their message, use codes to reference their integrations and then run a "campaign" by pressing "send". The integration being used is displayed at the top. A "campaign" grabs the top ranking users which can be seen on the "followers" page. It filters out all users who have been sent a DM in any prior campaigns, then it starts grabbing batches of the remaining ranked users (starting with those ranked the highest) to verify if they are at that moment followers. This process continues until enough users are found to satisfy the campaign size requested by the influencer and a batch of DMs are sent out with the influencer's message. All these campaigns are stored in the "campaigns.json" file. DM messages that have been sent out have their metadata stored in a "messages.json" file.

Note: there Californian laws to be aware of when automating direct messages.

To test the functionality of campaigns, a "dry-run" may be initiated. This mode will mimic the process of sending out DMs and generate output to files, but won't actually send out any DMs. You can then check the files to ensure the right people would receive a DM. If you then disable the dry-run option and hit "send" the dry-run campaigns will be deleted from the "campaigns.json" file and the messages will be sent out.

Settings

On the "Settings" page influencers are able to set integrations which can be referenced through special syntax on the "Campaigns" page.

Default Output Directories:

For MacOS you should find the data at:
~/Libary/Application Support/bt
For Windows you should find the data at:
C:\Users<user>\AppData\Roaming\bt

Future Plans:

We envision Better Twitter to have more features, but could not complete them all within the current deadline.

bt bounty project specfic plans

  • Automated engagement building and campaigns. Influencers would only need to load up the app and it will run a scan and processes a campaign every interval of time (such as 15 minutes).
  • More complex ranking formulas that are set by the user and can factor in time intervals of retweets and mentions. (For example 7 retweets done within an hour mean a user is more engaged than if they have 7 retweets done all time)
  • The ranked users listed on the "Followers" page arn't necessarily followers, the Twitter API GET friendships/lookup has a limit of 15 requests per 15 minutes and we want to only use it when confirming a user is a follower during a campaign. Displaying which users are followers can be done by using the GET followers/ids gateway but requires it's own unique process for accounts that have more than 75,000 followers as it would require waiting until all followers are pulled.
  • API Usage Page: We already log api requests and track the usage of them, but need a front-end page to display this information

@plutownium
Copy link

plutownium commented Jun 29, 2020

twitter-exporter

This is my submission to the Twitter Exporter contest.

I worked with a partner. Together we completed both the requirement for a local app and a hosted web site.

First, some features about the local app, which uses the "Mass DM" approach:

  • Followers are ranked and prioritized by account size.
  • You can close the app at anytime without losing track of who has already been messaged. The code picks up right where it left off using a .txt file as a database.
  • Approximately every 2 hours of running the app, the user receives a message with the expected remaining runtime.
  • The app sends 950 DMs per day, leaving its user with room to chat with friends. This can be adjusted to 990 or even 1000 DMs per day to truly optimize the length of time it takes to message all followers.
  • While it has a simple UI (runs in command prompt), it should satisfy the needs of any user looking to contact all their followers.
  • There is basic confirmation that the right message has been entered.

Here is an image of me using the compiled .exe file, the local app:

https://i.imgur.com/2FO15Ba.png

The script used to make the .exe and the .exe itself can be found here:

https://github.com/plutownium/twitterExporter

For our hosted website, we have additional features:

We analyze your followers to make your efforts count. We know what followers have the largest social media footprint, which are most loyal to you, and can help you leverage this inormation to build your brand. By targeting your most prominent "cadets" with give aways or exclusive material, you can make them more likely to give you exposure with mentions or retweets. By giving your loyal influencers access to insider information, you can reward them for sticking by your side and help them better represent you. Finally, you can DM the masses to keep them informed of whatever you want.

Our UI is rudimentary right now, but we have the backend to support advanced analytics to make your DMs count.

Here is a video using the client-side code for the website:

https://drive.google.com/file/d/15w7rIS0Zhh67yPT8rdbe47B2-Ha7U7Io/view

And this is the landing page followers will see when they click the included link:

https://cdn.discordapp.com/attachments/651923786903453702/722511234553085982/unknown.png

You can try the app yourself by going here, authorizing the app, and filling in the info. Click "send your followers a dm" after you get past authorization to open the dashboard

Here is the code for the hosted website:

https://github.com/Spade-and-Archer/MassDM

If our project is selected as the winner, please let me know via my Twitter handle, @rolypolyistaken

@sagardesai90
Copy link

sagardesai90 commented Jun 29, 2020

Mon Droit

An app that lets users DM their twitter followers en masse.

A bit of a background. Mon Droit is French for 'my right,' it comes from the motto of the English crown, 'dieu et mon droit, honi soit qui mal y pense' 'god and my right, shame on he who thinks evil of it.' It only felt right that those who have built their own empires on twitter have the right to take their followers to other locales (platforms), and perhaps see what comes of it.

6/30 Update: Added Dark Mode

Home

https://github.com/sagardesai90/mon-droit
MonDroit-Home

Dark Mode:
MonDroit-Home-Dark

Tokens Page

MonDroit-Tokens

Tokens Page - Dark Mode

MonDroit-Tokens-Dark

Input your tokens here, they are stored locally in a tokens.json file.

Campaigns Page

MonDroit-Campaign

Campaign Page - Dark Mode:

MonDroit-Campaign-Dark

History Page

MonDroit-History

History Page - Dark Mode

MonDroit-History-Dark

Set the twitter name you'd like to get a follower list for, that list is stored locally in the followers.json file. Then you can show that list in table form and select individual followers to send DMs. After selecting a subset of followers to DM, craft your message and send it out.

Limits

There is a rate limit of ~3000 followers being added to the followers.json every 15m.

You can send out 1000 DMs each day.

Future additions

  • Status updates on a particular campaign, especially useful for campaigns that may span multiple days (1000+ DMs)

  • Overall speed optimizations across the board

  • Will look into implementing saasify

@balajis
Copy link
Owner Author

balajis commented Jun 29, 2020

Wow! Review has begun.

So many awesome submissions by everyone. It will take us a little while to go through them, but we have begun the review process. Please feel free to reply with any questionsl

@Trudko
Copy link

Trudko commented Jun 30, 2020

Props to this initiative, this is twitter at its best.

@mspanish
Copy link

mspanish commented Jun 30, 2020

@balajis it's my assumption that you guys don't want people posting about updates to their apps, right? Am I correct that submissions have to stand on what was submitted before the deadline, and you'll be notifying people who've made the next round to then submit their updated versions?

@Trudko
Copy link

Trudko commented Jul 3, 2020

@balajis curious about this too, some apps were already updated.

@balajis
Copy link
Owner Author

balajis commented Jul 3, 2020

@mspanish @Trudko Updates to the apps are fine. We are setting up a systematic framework to score all of these. The goal is really to get the best open source program out there to the public, for free.

Divya Dhar Cohen (@dvd22dvd) and Ryan Shea (@shea256) are helping with the evaluation process. So if you get a note or email from them, don't be surprised :)

@dvd22dvd
Copy link

dvd22dvd commented Jul 4, 2020

Hey @plutownium @femyeda @socexporter @mrdavey @Trudko @npgeorge @socionity @syllogismos @thumpri @snowdot @petabite @JordanDworaczyk @IvanMontillaM @daisy1754 @mspanish @bconnorwhite @brunneis there wasn’t an email in your profile. Please email me at twitterexport@protonmail.com so that we can communicate with you.

@MohitKumar1991
Copy link

MohitKumar1991 commented Jul 5, 2020

@balajis @dvd22dvd

Here some updates for the third week for https://github.com/MohitKumar1991/twitter-export

  • Now supports one click deploy to heroku - so you get your own version of the tool deployed to heroku with one click
  • Supports gathering emails
  • Supports affiliate links
  • Supports sending DMs to any filtered followers

Very happy about this update because now anyone can use this by deploying to heroku. Total setup time is 10mins for heroku + twitter auth.

For the fourth week I will

  • Add some advanced filtering options
  • Make email submission page mobile responsive
  • Add audience analysis
  • Add multiple UI/UX improvements

@femyeda
Copy link

femyeda commented Jul 7, 2020

@dvd22dvd sent

@IvanMontillaM
Copy link

IvanMontillaM commented Jul 11, 2020

I missed it, so just sent! @dvd22dvd :-)

@MohitKumar1991
Copy link

MohitKumar1991 commented Jul 16, 2020

@balajis @dvd22dvd

Here some updates for the fourth week for https://github.com/MohitKumar1991/twitter-export
Now you can filter by occupation (founder, engineer, investor etc)
UI/UX Redo - much better looking and multiple confusing flows were simplified.
Couple of bug fixes

Very happy about this update as well because now it's quite usable.

For the fifth week I will fix some bugs and that's it. I am not proud of how much time I have spent on this by now.

@balajis
Copy link
Owner Author

balajis commented Jul 18, 2020

@MohitKumar1991

I am not proud of how much time I have spent on this by now.

Please be super proud! We have a bunch of volunteers reviewing apps, and @dvd22dvd should have gotten in touch with folks to get their payment info for the honorable mentions. With the Twitter hack your work on this project is more important than ever.

@mohit-sentieo
Copy link

mohit-sentieo commented Jul 18, 2020

@balajis Thanks for the words of encouragement. I did get her email and I have mailed her the new version as well as a heroku hosted version( in case she doesn't have a heroku account). Everything works on local anyways except for affiliate links.

Also I was wondering if you found anyone who is working on the ethereum subscriptions EIP that you shared some time back. Found it to be super interesting. Would love to collaborate on that.

@Trudko
Copy link

Trudko commented Jul 19, 2020

@balajis @dvd22dvd

Made few changes too:

  • Influencer can set own message, which will be seen by followers after clicking invite link.
  • Simplified UI bit more, added simple onboard after first login
  • fixed several issues in UI

Next plans

  • create proper landing page
  • improve onboarding process
  • support mobile
  • improve integration with Substack and Ghost. You can now export followers list to Ghost compatible csv. Would like to support Substack and maybe integrate Ghost Members API (which is experimental for now)
  • Add option to turn off Ethereum payouts. Now every follower sees Ethereum address input which is optional. I don't think every influencer wants to payout affiliates.
  • thinking about implementing ideas from @balajis talks from Blockstack conference. Let me know what do you think.

@mohit-sentieo
Copy link

mohit-sentieo commented Jul 21, 2020

@balajis @dvd22dvd
For the next week, I will be adding the following features requested by some folks who I am trying to get to use this.

  • Updating followers every day
  • Knowing which followers left, which were newly added
  • Knowing which followers changed their descriptions or something else
  • Fetching twitter followers for someone else
  • Detecting possible fake twitter users (Probably use someone else's model)

Just wondering if you have any comments or specific requirements.

@Trudko
Copy link

Trudko commented Jul 23, 2020

  • we fixed basic responsivity issues, although still need to implement final touches
  • building proper landing page
  • spoke with John Nolan from Ghost, super friendly super helpful!
  • will be adding ability to turn of Ethereum rewards. Score will be still kept so community leader can still decide how to reward community members.
  • I want to add local encryption for followers data so if service is run on cloud, your data will be stored encrypted on the server. We need privacy first apps.
  • for now we aren't planning new functionality, but rather ask Twitter power users for the feedback and improve what we have right now.

@dvd22dvd
Copy link

dvd22dvd commented Jul 25, 2020

Hi @daisy1754 @shea256 @snowdot @moesalih @femyeda @kgkrishnavbi @CodeHownd @transitive-bullshit @vasa-develop @treylorswift @martonlanga @plutownium @mrdavey @MohitKumar1991 @neelsomani @ShreyasJothish @socionity @DeveloperRyan @sandeepvellanki @mspanish @brunneis @socexporter @sagardesai90 @npgeorge @JordanDworaczyk @IvanMontillaM @bconnorwhite @steveblackmon @DennisRutjes @gg2001 @thumpri @kanav99 @tksumanth1994 @kira13zyh @petabite @syllogismos @Trudko @charliepyle

Thanks for your submissions! We have completed an initial pass over all the submissions. We'll be doing a final pass over the next couple of weeks. Could you please ensure your services are online till Monday 10th August? Feel free to update your post to make it easy for the reviewer to access everything they'll need. Unfortunately, if we aren't able to access your service, we won't be able to review it.

Finally, if you haven't sent your BTC address to me yet, please do so at twitterexport@protonmail.com Thank you!

cc @balajis

@Trudko
Copy link

Trudko commented Jul 27, 2020

Quick update:

  • much better responsivity on mobile
  • we have new landing page
  • you can now turn off ethereum payouts, which hides ethereum address for followers too. Useful if you have non crypto based incentives only.
  • gathering feedback from multiple possible users, including people who contributed to the bounty
  • researched possibilities for data encryption, metamask recently has added ability to encrypt / decrypt data . So far looks good, but UX is still lacking so will wait or have to implement own solution

Next steps:

  • add basic filters for followers data
  • improve onboarding for people not familiar with the bounty
  • improve email verification. Right now it's easy export followers list and upload them to https://kickbox.com/ for verification. It should be up to user what service to use for verification before uploading the list to Ghost

@Trudko
Copy link

Trudko commented Aug 4, 2020

Few significant updates.

Improved onboarding

  • we've added simple onboard which helps Twitter owner with basic settings. It's ongoing work, so we'll update based on customer interviews.
    Kapture 2020-08-04 at 10 04 11

Added email verification

  • each follower has to already be follower of the main account and now we added mandatory email verification. We've integrated Postmark, which has great pricing, nice dashboard ofc, it's easy to switch to something else.
  • if you referrer someone, you only get points when they verify email and Twitter account

Screenshot 2020-08-04 at 12 41 11

Filtering based on number of followers

  • we still use # of followers as a proxy for value of the follower, so now you are able to filter follower based on min. of followers and export only results

number_of_followers

Other

  • ofc several bug fixes

Next steps

  • still focused on getting the basic use case twitter -> export CSV -> ghost / substack import
  • started to speaking with several smaller and bigger accounts for a feedback

CC @dvd22dvd

@snowdot
Copy link

snowdot commented Aug 23, 2020

Hello @balajis, @dvd22dvd,

Please let me know if it still is mandatory to ensure that our services are online.
Also, could you please give us an estimated date for the final results?

Thank you in advance!

@Trudko
Copy link

Trudko commented Sep 21, 2020

Any updates please?

@IvanMontillaM
Copy link

IvanMontillaM commented Oct 12, 2020

Hello,

Is there any updates? We are concerned this hasn't moved on a while.

@snowdot
Copy link

snowdot commented Oct 27, 2020

Sir @balajis,

We are all programmer here and we need closure (pun intended) regarding this bounty.
Is it too much to ask for the results of this considering the hard work that all of us put in?

Also, I'll have to take my application down since I'm paying for hosting it for some months now and there is zero traffic on it which makes me believe that the review is over..

Much appreciated!

@balajis
Copy link
Owner Author

balajis commented Nov 11, 2020

Hey folks, we finally got through all the applications. We have a winner and it's socialexporter.com by @moesalih_. Will be putting up an announcement soon. If you have an existing application that was up for a while during evaluation and it cost some money in server costs, send me the receipts at balajis at balajis and I'll see what I can do.

We may also award some (more) runner-up prizes for other folks, in addition to the honoraria we already paid out.
@snowdot @IvanMontillaM @Trudko

@syllogismos
Copy link

syllogismos commented Nov 13, 2020

Congrats to the winners @moesalih . It was really fun participating in this competition.

@Trudko
Copy link

Trudko commented Nov 13, 2020

Great, anyone having a plan to continue work on this?

@ZilvinasKucinskas
Copy link

ZilvinasKucinskas commented Feb 23, 2021

Hey everyone, I am running ExportData.

I already have 200M+ Twitter users in the DB, trends data dated from 2019, lots of tweets, and no performance problems with the infrastructure I have. However, I need help with developing new features.

If somebody wants to continue working with the Twitter API on something a little bit different, hit me up via email (you can find it in my Github profile) and we can chat about it. We might find different collaboration angles - PT or FT work, profit sharing, etc.

What we might work next

  • Influencer search (search across 200M+ users by keyword in bio, location, filter users having contact information publicly present, etc..) - similar to FollowerWonk bio search
  • Tweet monitoring by hashtag or keyword and their analytics
  • Account analytics - something such as Ilo.so
  • Displaying top Twitter trends across the globe (current, longest trending, etc..)

Stack

  • API (97.9% test coverage): Ruby, Rails, PostgreSQL, ElasticSearch
  • Frontend: React.JS, MobX
  • Blog: Ghost
  • DevOps: Terraform, Docker, AWS
  • Other: Stripe, Freshchat, LogRocket, NewRelic, Notion, Trello, Slack and similar

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests