Twitter bot for generating photo descriptions

According to Twitter support bot account has been suspended due to violations of rules:

• Creating serial and/or multiple accounts with overlapping uses
• Evading a permanent suspension by creating or using another account
• Cross-posting Tweets or links across multiple accounts
• Aggressive following, particularly through automated means

---

Twitter: https://twitter.com/GramtionBot

Source Code: https://github.com/lRomul/gramtion

---

This repo contains the source code of the Twitter @GramtionBot for generating photo descriptions.
Use cases and intends:

• Help visually impaired Twitter users. Good image descriptions (alt text) will help them understand what is happening in an image. Instagram and Facebook use deep learning for image captioning. Twitter users can only add custom alt text descriptions themselves. Automation of alt text generation will help Twitter be more accessible.
• Collect dataset for image captioning (legal issues for this use case are yet to be discussed). Annotations can be done by creating polls about prediction quality and getting corrected descriptions from users. Twitter API v1.1 has not the ability to create polls, but it will be added in API v2.

How to use

Tweet photo with mention @GramtionBot or reply with mention to a tweet with a photo and the bot will send you an auto-generated image description.

Dependencies

Gramtion is mainly built from ready-to-use third party libraries:

• Image captioning model taken from self-critical.pytorch.
• Evaluate text and image similarity with CLIP by OpenAI.
• OCR and image labels by Google Vision Ai.
• Bot written with Tweepy.
• Configuration settings implemented with pydantic.
• Docker image based on Dokai.

Current issues

• Some descriptions may be confusing. Annotations may be created by using polls about prediction quality and getting corrected descriptions from users. Twitter API v1.1 has not the ability to create polls, but it will be added in API v2 endpoint POST /2/tweets.
• For drawings and some other types of images, the predictions are pretty random.
• Some results may reflect inherent gender and racial biases of open datasets.

Run own bot

To run an instance of the bot you need to install Docker and create Twitter API auth credentials.
If you have a Twitter developer account, but don't want to use it as a bot account, you can authenticate a new user that’s not has a developer account with twurl.

• Create .env file with credentials.

  CONSUMER_KEY={{ consumer_key }}
  CONSUMER_SECRET={{ consumer_secret }}
  ACCESS_TOKEN={{ access_token }}
  ACCESS_TOKEN_SECRET={{ access_token_secret }}
  

• Setup Google Vision AI and create account key (link). Copy key as google_key.json .

• Run Docker container with running the bot

  docker run -d --restart=always \
      --env-file .env \
      -v google_key.json:/workdir/google_key.json \
      --name=gramtion \
      ghcr.io/lromul/gramtion:0.0.5

• Open logs

  docker logs -f gramtion

• Stop container

  docker stop gramtion
  docker rm gramtion