Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scrape images, video, and post forwarding information for Telegram #413

Open
wants to merge 30 commits into
base: master
Choose a base branch
from
Open

Scrape images, video, and post forwarding information for Telegram #413

wants to merge 30 commits into from

Commits on Feb 24, 2022

  1. Scrape images, video, and post forwarding information for Telegram ch… 

    …annel posts
    @loganwilliams
    loganwilliams committed Feb 24, 2022
    Configuration menu
    Copy the full SHA
    72b26f2 View commit details
    Browse the repository at this point in the history

  2. Fix KeyError caused by retweets without URLs in TwitterProfileScraper

    @loganwilliams
    loganwilliams committed Feb 24, 2022
    Configuration menu
    Copy the full SHA
    de4ebed View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2022

  1. Clean up unnecessary imports

    @loganwilliams
    loganwilliams committed Mar 8, 2022
    Configuration menu
    Copy the full SHA
    b8efce2 View commit details
    Browse the repository at this point in the history

Commits on Mar 29, 2022

  1. added capability to extract the number of channel members when the th… 

    …e string in membersDiv has the word 'subscribers' rather than 'members'.
    @trislee
    trislee committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    ed82916 View commit details
    Browse the repository at this point in the history

  2. handled case where channel has no profile image

    @trislee
    trislee committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    fb8d73a View commit details
    Browse the repository at this point in the history

Commits on Mar 30, 2022

  1. added capability to scrape multiple videos from a single post

    @trislee
    trislee committed Mar 30, 2022
    Configuration menu
    Copy the full SHA
    d32c9ad View commit details
    Browse the repository at this point in the history

Commits on Mar 31, 2022

  1. implemented Media dataclasses for Telegram, and added variable for ex… 

    …tracting a post's view count
    @trislee
    trislee committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    a7eb54d View commit details
    Browse the repository at this point in the history

  2. added a forwardedUrl attribute to TelegramPost and made forwarded att… 

    …ribute type Channel.
    @trislee
    trislee committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    4e59638 View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2022

  1. fixed edge case for videos that have data-link-attr but no href attri… 

    …bute
    @trislee
    trislee committed Apr 3, 2022
    Configuration menu
    Copy the full SHA
    2ce014a View commit details
    Browse the repository at this point in the history

  2. Merge branch 'JustAnotherArchivist:master' into master

    @trislee
    trislee committed Apr 3, 2022
    Configuration menu
    Copy the full SHA
    f978954 View commit details
    Browse the repository at this point in the history

Commits on Apr 17, 2022

  1. made Telegram scraper not return full channel info for forwarded_from… 

    … attribute; fixed video edge cases.
    @trislee
    trislee committed Apr 17, 2022
    Configuration menu
    Copy the full SHA
    babcddd View commit details
    Browse the repository at this point in the history

  2. fixed issue where Telegram scraper terminated early because some page… 

    …s didn't have a next page link (added reasonable default)
    @trislee
    trislee committed Apr 17, 2022
    Configuration menu
    Copy the full SHA
    1e4e0c2 View commit details
    Browse the repository at this point in the history

  3. fixed issue where some videos and photos weren't being scraped (becau… 

    …se they weren't in a post containing a 'tgme_widget_message_text' div
    @trislee
    trislee committed Apr 17, 2022
    Configuration menu
    Copy the full SHA
    b276c3c View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2022

  1. added additional termination criteria to Telegram scraper

    @trislee
    trislee committed Apr 21, 2022
    Configuration menu
    Copy the full SHA
    97d38e5 View commit details
    Browse the repository at this point in the history

  2. added additional attributes for hashtags and user mentions, removed r… 

    …edundant outlinks
    @trislee
    trislee committed Apr 21, 2022
    Configuration menu
    Copy the full SHA
    9b3faec View commit details
    Browse the repository at this point in the history

  3. moved forward finding out of tgme_widget_message_text clause, since i… 

    …t wasn't correctly getting the forwarding information in forwarded posts that contained attachments but no text
    @trislee
    trislee committed Apr 21, 2022
    Configuration menu
    Copy the full SHA
    21f7b62 View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2022

  1. improved consistency of code formatting and added _STYLE_MEDIA_URL_PA… 

    …TTERN as variable
    @trislee
    trislee committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    5648e95 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2022

  1. Merge branch 'master' into telegram-media

    @trislee
    trislee committed May 9, 2022
    Configuration menu
    Copy the full SHA
    c18ca0f View commit details
    Browse the repository at this point in the history

  2. Merge pull request #2 from bellingcat/telegram-media 

    Implemented JustAnotherArchivist's requested changes to Telegram scraper from PR
    @trislee
    trislee committed May 9, 2022
    Configuration menu
    Copy the full SHA
    0a4bd39 View commit details
    Browse the repository at this point in the history

  3. fixed merge conflicts

    @trislee
    trislee committed May 9, 2022
    Configuration menu
    Copy the full SHA
    f385135 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'JustAnotherArchivist-master'

    @trislee
    trislee committed May 9, 2022
    Configuration menu
    Copy the full SHA
    b13e62e View commit details
    Browse the repository at this point in the history

  5. forgot to save modified twitter.py module

    @trislee
    trislee committed May 9, 2022
    Configuration menu
    Copy the full SHA
    e2d9223 View commit details
    Browse the repository at this point in the history

Commits on May 25, 2022

  1. Merge pull request #4 from JustAnotherArchivist/master 

    upstream merge
    @trislee
    trislee committed May 25, 2022
    Configuration menu
    Copy the full SHA
    0822a9c View commit details
    Browse the repository at this point in the history

  2. merged master into more-tg-info to update upstream PR

    @trislee
    trislee committed May 25, 2022
    Configuration menu
    Copy the full SHA
    07a5f6f View commit details
    Browse the repository at this point in the history

  3. fixed merge

    @trislee
    trislee committed May 25, 2022
    Configuration menu
    Copy the full SHA
    65723f1 View commit details
    Browse the repository at this point in the history

Commits on Jun 23, 2022

  1. fixed typo

    @trislee
    trislee committed Jun 23, 2022
    Configuration menu
    Copy the full SHA
    56e4232 View commit details
    Browse the repository at this point in the history

  2. incorporated requested changes from maintainer, removed modifications… 

    … to VK module
    @trislee
    trislee committed Jun 23, 2022
    Configuration menu
    Copy the full SHA
    056cd62 View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2022

  1. fixed edge case where channel with no members fails _get_entity

    @trislee
    trislee committed Jul 5, 2022
    Configuration menu
    Copy the full SHA
    73f10a4 View commit details
    Browse the repository at this point in the history

Commits on Nov 30, 2022

  1. fixed edge case where members information wasnt included

    @trislee
    trislee committed Nov 30, 2022
    Configuration menu
    Copy the full SHA
    cbdfeed View commit details
    Browse the repository at this point in the history

Commits on Apr 4, 2023

  1. merged upstram changes

    @trislee
    trislee committed Apr 4, 2023
    Configuration menu
    Copy the full SHA
    cacd783 View commit details
    Browse the repository at this point in the history