Out of Context Image Tweets + Community Notes Dataset

Dataset Description

Tweet info

• 'id': Unique identifier for the tweet.
• 'text': The text content of the tweet.
• 'date': The date and time when the tweet was created.
• 'author_id': Unique identifier for the author of the tweet.
• 'author_username': The username of the author of the tweet.
• 'retweet_count': The number of retweets the tweet has received.
• 'hashtags': A list of hashtags used in the tweet.
• 'mentions': A list of users mentioned in the tweet.
• 'image_urls': A list of URLs to images included in the tweet.
• 'tweet_url': The URL to the tweet.

Community Note info

Twitter community note documentation

Note Info

• 'noteId': Unique identifier for the community note.
• 'noteAuthorParticipantId': Unique identifier for the author of the community note.
• 'createdAtMillis': The date and time when the community note was created.
• 'tweetId': Unique identifier for the tweet associated with the community note.
• 'classification': The author's classification of the tweet (e.g. "MISINFORMED_OR_POTENTIALLY_MISLEADING")
• 'misleading mechanism': A series of 1 or 0 values indicating the mechanisms of misleading information (e.g. "misleadingOther", "misleadingFactualError" "misleadingManipulatedMedia", "misleadingOutdatedInformation", etc.).
• 'trustworthySources': Does the community note author believe they used trustworthy sources? (e.g. 1 or 0)
• 'summary': A summary of the community note.
• 'isMediaNote': Indicates whether the community note is a media note (e.g. 1 or 0).

Note Status History

User Enrollment

Community Note Filters

We did not keep the entire community note dataset. Instead we applied the following filters to get notes more relevant to the out of context image task:

1. community_note.classification: We only kept notes where the classification isn't "NOT_MISLEADING". If the tweet isn't misleading, then it's not relevant to our task.
2. community_note.isMediaNote: We only kept notes where the note is a media note i.e. == 1. The tweets must contain an image that we can perform reverse image search on later.
3. community_note.misleadingMissingImportantContext: We only kept notes where "misleadingMissingImportantContext" is reported as one of the misleading mechanisms. We want to avoid notes that are misleading for other reasons (e.g. "misleadingFactualError", "misleadingOutdatedInformation", etc.) and focus on the out of context image task.
4. community_note.misleadingManipulatedMedia: We only kept notes where "misleadingManipulatedMedia" is NOT reported as one of the misleading mechanisms. We want to avoid notes about digitally altered images in favor of notes with original images being taken out of context.
5. community_note.summary Keywords: We only keep notes with at least one of the following keywords in the summary: ["photo", "image", "photograph", "picture", "photos", "images", "photographs", "pictures"]. The keywords help to ensure that the community note is about an image in the tweet.

Generated Features

• 'llm_image_classification': The classification of the image in the tweet generated by an LLM. The classification is one of "misleading" or "contextual".

  • "misleading": The image itself is misleading. e.g. An image with embedded text making a false claim.
  • "contextual": The image itself not misleading, but can be taken out of context by the tweet. e.g. An image of a protest in a different country than the one the tweet is about.

• 'topical categories': We prompt the LLM to generate a list of categories relevant to the given community note. The full list of categories and their descriptions can be found in this prompts file.

• 'reverse_image_search_results': Contains the results of a reverse image search via Google Web Entity API. We keep the top 3 results for each image.

  • 'page_url': The URL of the page where the image was found.
  • 'title': The title of the page where the image was found.
  • 'partial_matching_images': A list of URLs to images that partially match the image in the tweet.
  • 'full_matching_images': A list of URLs to images that fully match the image in the tweet.
  • 'image_context': Information from performing a Google search on the 'page_url'. Most text is truncated depending on how much Google shows.
    • 'title': The title of the page (usually longer than the 'title' field above).
    • 'link': The URL of the page, usually the same as 'page_url'.
    • 'snippet': A snippet of text from the page.

• 'dememe_reverse_image_search_results`: Contains the results of a reverse image search using a 'dememed' version of the image. The 'dememed' image has it's embedded text cropped out to improve search results. We keep the top 3 in the same format as above.