Skip to content

feat: extend the query search to get more details #26

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 12 commits into from
Feb 19, 2025
Merged

feat: extend the query search to get more details #26

merged 12 commits into from
Feb 19, 2025

Conversation

alvin-reyes
Copy link
Contributor

@alvin-reyes alvin-reyes commented Feb 19, 2025 •
edited
Loading

Changes

Don't merge yet, I still need to test using TWITTER_ACCOUNTS. I can't seem to find a working account.

  • refactor to get more information from the twitter X api
  • instead just returning the tweet, we now return the tweet result which includes all the data

Dependent on

https://github.com/masa-finance/tee-indexer/pull/45

Verification

Ran with the following

// my own script that calls the worker only
./tee-worker-job.sh -q "what is trending?" -m 10
Configuration:
API URL: http://157.180.2.172:8083
Query: what is trending?
Max Results: 10

Generating job...
Job generated successfully
Submitting job...
Job submitted successfully. UUID: a4a1bc25-74b7-4d1b-9a06-0fe1cbe4839c
Checking job status...
Got job status
Decrypting results...
Results:
[
  {
    "Tweet": null,
    "TweetXData": {
      "author_id": "489069983",
      "entities": {
        "urls": null,
        "annotations": [
          {
            "start": 72,
            "end": 79,
            "probability": 0.6225,
            "type": "Person",
            "normalized_text": "FAYEYOKO"
          },
          {
            "start": 102,
            "end": 114,
            "probability": 0.5536,
            "type": "Other",
            "normalized_text": "PromiseToStay"
          }
        ]
      },
      "id": "1892039425959346193",
      "possibly_sensitive": false,
      "reply_settings": "everyone",
      "conversation_id": "1892039425959346193",
      "public_metrics": {
        "retweet_count": 120,
        "reply_count": 0,
        "like_count": 0,
        "quote_count": 0,
        "bookmark_count": 0,
        "impression_count": 0
      },
      "edit_controls": {
        "edits_remaining": 5,
        "is_edit_eligible": false,
        "editable_until": "2025-02-19T03:32:06Z"
      },
      "text": "RT @chakrulo321: Who is here for trending? RT and Reply \"ALWAYS HER FOR FAYEYOKO\"\n\nFY NO MATTER WHAT\n#PromiseToStay",
      "context_annotations": null,
      "created_at": "2025-02-19T02:32:06Z",
      "display_text_range": [
        0,
        115
      ],
      "lang": "en",
      "edit_history_tweet_ids": [
        "1892039425959346193"
      ],
      "referenced_tweets": [
        {
          "type": "retweeted",
          "id": "1892024895585804295"
        }
      ]
    },
    "TweetXMeta": {
      "newest_id": "1892040265302528116",
      "oldest_id": "1892039425959346193",
      "result_count": 10
    },
    "ThreadCursor": null,
    "Error": null
  },
  {
    "Tweet": null,
    "TweetXData": {
      "author_id": "489069983",
      "entities": {
        "urls": null,
        "annotations": [
          {
            "start": 72,
            "end": 79,
            "probability": 0.6225,
            "type": "Person",
            "normalized_text": "FAYEYOKO"
          },
          {
            "start": 102,
            "end": 114,
            "probability": 0.5536,
            "type": "Other",
            "normalized_text": "PromiseToStay"
          }
        ]
      },
      "id": "1892039425959346193",
      "possibly_sensitive": false,
      "reply_settings": "everyone",
      "conversation_id": "1892039425959346193",
      "public_metrics": {
        "retweet_count": 120,
        "reply_count": 0,
        "like_count": 0,
        "quote_count": 0,
        "bookmark_count": 0,
        "impression_count": 0
      },
      "edit_controls": {
        "edits_remaining": 5,
        "is_edit_eligible": false,
        "editable_until": "2025-02-19T03:32:06Z"
      },
      "text": "RT @chakrulo321: Who is here for trending? RT and Reply \"ALWAYS HER FOR FAYEYOKO\"\n\nFY NO MATTER WHAT\n#PromiseToStay",
      "context_annotations": null,
      "created_at": "2025-02-19T02:32:06Z",
      "display_text_range": [
        0,
        115
      ],
      "lang": "en",
      "edit_history_tweet_ids": [
        "1892039425959346193"
      ],
      "referenced_tweets": [
        {
          "type": "retweeted",
          "id": "1892024895585804295"
        }
      ]
    },
    "TweetXMeta": {
      "newest_id": "1892040265302528116",
      "oldest_id": "1892039425959346193",
      "result_count": 10
    },
    "ThreadCursor": null,
    "Error": null
  },
  {
    "Tweet": null,
    "TweetXData": {
      "author_id": "489069983",
      "entities": {
        "urls": null,
        "annotations": [
          {
            "start": 72,
            "end": 79,
            "probability": 0.6225,
            "type": "Person",
            "normalized_text": "FAYEYOKO"
          },
          {
            "start": 102,
            "end": 114,
            "probability": 0.5536,
            "type": "Other",
            "normalized_text": "PromiseToStay"
          }
        ]
      },
      "id": "1892039425959346193",
      "possibly_sensitive": false,
      "reply_settings": "everyone",
      "conversation_id": "1892039425959346193",
      "public_metrics": {
        "retweet_count": 120,
        "reply_count": 0,
        "like_count": 0,
        "quote_count": 0,
        "bookmark_count": 0,
        "impression_count": 0
      },
      "edit_controls": {
        "edits_remaining": 5,
        "is_edit_eligible": false,
        "editable_until": "2025-02-19T03:32:06Z"
      },
      "text": "RT @chakrulo321: Who is here for trending? RT and Reply \"ALWAYS HER FOR FAYEYOKO\"\n\nFY NO MATTER WHAT\n#PromiseToStay",
      "context_annotations": null,
      "created_at": "2025-02-19T02:32:06Z",
      "display_text_range": [
        0,
        115
      ],
      "lang": "en",
      "edit_history_tweet_ids": [
        "1892039425959346193"
      ],
      "referenced_tweets": [
        {
          "type": "retweeted",
          "id": "1892024895585804295"
        }
      ]
    },
    "TweetXMeta": {
      "newest_id": "1892040265302528116",
      "oldest_id": "1892039425959346193",
      "result_count": 10
    },
    "ThreadCursor": null,
    "Error": null
  },
  {
    "Tweet": null,
    "TweetXData": {
      "author_id": "489069983",
      "entities": {
        "urls": null,
        "annotations": [
          {
            "start": 72,
            "end": 79,
            "probability": 0.6225,
            "type": "Person",
            "normalized_text": "FAYEYOKO"
          },
          {
            "start": 102,
            "end": 114,
            "probability": 0.5536,
            "type": "Other",
            "normalized_text": "PromiseToStay"
          }
        ]
      },
      "id": "1892039425959346193",
      "possibly_sensitive": false,
      "reply_settings": "everyone",
      "conversation_id": "1892039425959346193",
      "public_metrics": {
        "retweet_count": 120,
        "reply_count": 0,
        "like_count": 0,
        "quote_count": 0,
        "bookmark_count": 0,
        "impression_count": 0
      },
      "edit_controls": {
        "edits_remaining": 5,
        "is_edit_eligible": false,
        "editable_until": "2025-02-19T03:32:06Z"
      },
      "text": "RT @chakrulo321: Who is here for trending? RT and Reply \"ALWAYS HER FOR FAYEYOKO\"\n\nFY NO MATTER WHAT\n#PromiseToStay",
      "context_annotations": null,
      "created_at": "2025-02-19T02:32:06Z",
      "display_text_range": [
        0,
        115
      ],
      "lang": "en",
      "edit_history_tweet_ids": [
        "1892039425959346193"
      ],
      "referenced_tweets": [
        {
          "type": "retweeted",
          "id": "1892024895585804295"
        }
      ]
    },
    "TweetXMeta": {
      "newest_id": "1892040265302528116",
      "oldest_id": "1892039425959346193",
      "result_count": 10
    },
    "ThreadCursor": null,
    "Error": null
  },
  {
    "Tweet": null,
    "TweetXData": {
      "author_id": "489069983",
      "entities": {
        "urls": null,
        "annotations": [
          {
            "start": 72,
            "end": 79,
            "probability": 0.6225,
            "type": "Person",
            "normalized_text": "FAYEYOKO"
          },
          {
            "start": 102,
            "end": 114,
            "probability": 0.5536,
            "type": "Other",
            "normalized_text": "PromiseToStay"
          }
        ]
      },
      "id": "1892039425959346193",
      "possibly_sensitive": false,
      "reply_settings": "everyone",
      "conversation_id": "1892039425959346193",
      "public_metrics": {
        "retweet_count": 120,
        "reply_count": 0,
        "like_count": 0,
        "quote_count": 0,
        "bookmark_count": 0,
        "impression_count": 0
      },
      "edit_controls": {
        "edits_remaining": 5,
        "is_edit_eligible": false,
        "editable_until": "2025-02-19T03:32:06Z"
      },
      "text": "RT @chakrulo321: Who is here for trending? RT and Reply \"ALWAYS HER FOR FAYEYOKO\"\n\nFY NO MATTER WHAT\n#PromiseToStay",
      "context_annotations": null,
      "created_at": "2025-02-19T02:32:06Z",
      "display_text_range": [
        0,
        115
      ],
      "lang": "en",
      "edit_history_tweet_ids": [
        "1892039425959346193"
      ],
      "referenced_tweets": [
        {
          "type": "retweeted",
          "id": "1892024895585804295"
        }
      ]
    },
    "TweetXMeta": {
      "newest_id": "1892040265302528116",
      "oldest_id": "1892039425959346193",
      "result_count": 10
    },
    "ThreadCursor": null,
    "Error": null
  },
  {
    "Tweet": null,
    "TweetXData": {
      "author_id": "489069983",
      "entities": {
        "urls": null,
        "annotations": [
          {
            "start": 72,
            "end": 79,
            "probability": 0.6225,
            "type": "Person",
            "normalized_text": "FAYEYOKO"
          },
          {
            "start": 102,
            "end": 114,
            "probability": 0.5536,
            "type": "Other",
            "normalized_text": "PromiseToStay"
          }
        ]
      },
      "id": "1892039425959346193",
      "possibly_sensitive": false,
      "reply_settings": "everyone",
      "conversation_id": "1892039425959346193",
      "public_metrics": {
        "retweet_count": 120,
        "reply_count": 0,
        "like_count": 0,
        "quote_count": 0,
        "bookmark_count": 0,
        "impression_count": 0
      },
      "edit_controls": {
        "edits_remaining": 5,
        "is_edit_eligible": false,
        "editable_until": "2025-02-19T03:32:06Z"
      },
      "text": "RT @chakrulo321: Who is here for trending? RT and Reply \"ALWAYS HER FOR FAYEYOKO\"\n\nFY NO MATTER WHAT\n#PromiseToStay",
      "context_annotations": null,
      "created_at": "2025-02-19T02:32:06Z",
      "display_text_range": [
        0,
        115
      ],
      "lang": "en",
      "edit_history_tweet_ids": [
        "1892039425959346193"
      ],
      "referenced_tweets": [
        {
          "type": "retweeted",
          "id": "1892024895585804295"
        }
      ]
    },
    "TweetXMeta": {
      "newest_id": "1892040265302528116",
      "oldest_id": "1892039425959346193",
      "result_count": 10
    },
    "ThreadCursor": null,
    "Error": null
  },
  {
    "Tweet": null,
    "TweetXData": {
      "author_id": "489069983",
      "entities": {
        "urls": null,
        "annotations": [
          {
            "start": 72,
            "end": 79,
            "probability": 0.6225,
            "type": "Person",
            "normalized_text": "FAYEYOKO"
          },
          {
            "start": 102,
            "end": 114,
            "probability": 0.5536,
            "type": "Other",
            "normalized_text": "PromiseToStay"
          }
        ]
      },
      "id": "1892039425959346193",
      "possibly_sensitive": false,
      "reply_settings": "everyone",
      "conversation_id": "1892039425959346193",
      "public_metrics": {
        "retweet_count": 120,
        "reply_count": 0,
        "like_count": 0,
        "quote_count": 0,
        "bookmark_count": 0,
        "impression_count": 0
      },
      "edit_controls": {
        "edits_remaining": 5,
        "is_edit_eligible": false,
        "editable_until": "2025-02-19T03:32:06Z"
      },
      "text": "RT @chakrulo321: Who is here for trending? RT and Reply \"ALWAYS HER FOR FAYEYOKO\"\n\nFY NO MATTER WHAT\n#PromiseToStay",
      "context_annotations": null,
      "created_at": "2025-02-19T02:32:06Z",
      "display_text_range": [
        0,
        115
      ],
      "lang": "en",
      "edit_history_tweet_ids": [
        "1892039425959346193"
      ],
      "referenced_tweets": [
        {
          "type": "retweeted",
          "id": "1892024895585804295"
        }
      ]
    },
    "TweetXMeta": {
      "newest_id": "1892040265302528116",
      "oldest_id": "1892039425959346193",
      "result_count": 10
    },
    "ThreadCursor": null,
    "Error": null
  },
  {
    "Tweet": null,
    "TweetXData": {
      "author_id": "489069983",
      "entities": {
        "urls": null,
        "annotations": [
          {
            "start": 72,
            "end": 79,
            "probability": 0.6225,
            "type": "Person",
            "normalized_text": "FAYEYOKO"
          },
          {
            "start": 102,
            "end": 114,
            "probability": 0.5536,
            "type": "Other",
            "normalized_text": "PromiseToStay"
          }
        ]
      },
      "id": "1892039425959346193",
      "possibly_sensitive": false,
      "reply_settings": "everyone",
      "conversation_id": "1892039425959346193",
      "public_metrics": {
        "retweet_count": 120,
        "reply_count": 0,
        "like_count": 0,
        "quote_count": 0,
        "bookmark_count": 0,
        "impression_count": 0
      },
      "edit_controls": {
        "edits_remaining": 5,
        "is_edit_eligible": false,
        "editable_until": "2025-02-19T03:32:06Z"
      },
      "text": "RT @chakrulo321: Who is here for trending? RT and Reply \"ALWAYS HER FOR FAYEYOKO\"\n\nFY NO MATTER WHAT\n#PromiseToStay",
      "context_annotations": null,
      "created_at": "2025-02-19T02:32:06Z",
      "display_text_range": [
        0,
        115
      ],
      "lang": "en",
      "edit_history_tweet_ids": [
        "1892039425959346193"
      ],
      "referenced_tweets": [
        {
          "type": "retweeted",
          "id": "1892024895585804295"
        }
      ]
    },
    "TweetXMeta": {
      "newest_id": "1892040265302528116",
      "oldest_id": "1892039425959346193",
      "result_count": 10
    },
    "ThreadCursor": null,
    "Error": null
  },
  {
    "Tweet": null,
    "TweetXData": {
      "author_id": "489069983",
      "entities": {
        "urls": null,
        "annotations": [
          {
            "start": 72,
            "end": 79,
            "probability": 0.6225,
            "type": "Person",
            "normalized_text": "FAYEYOKO"
          },
          {
            "start": 102,
            "end": 114,
            "probability": 0.5536,
            "type": "Other",
            "normalized_text": "PromiseToStay"
          }
        ]
      },
      "id": "1892039425959346193",
      "possibly_sensitive": false,
      "reply_settings": "everyone",
      "conversation_id": "1892039425959346193",
      "public_metrics": {
        "retweet_count": 120,
        "reply_count": 0,
        "like_count": 0,
        "quote_count": 0,
        "bookmark_count": 0,
        "impression_count": 0
      },
      "edit_controls": {
        "edits_remaining": 5,
        "is_edit_eligible": false,
        "editable_until": "2025-02-19T03:32:06Z"
      },
      "text": "RT @chakrulo321: Who is here for trending? RT and Reply \"ALWAYS HER FOR FAYEYOKO\"\n\nFY NO MATTER WHAT\n#PromiseToStay",
      "context_annotations": null,
      "created_at": "2025-02-19T02:32:06Z",
      "display_text_range": [
        0,
        115
      ],
      "lang": "en",
      "edit_history_tweet_ids": [
        "1892039425959346193"
      ],
      "referenced_tweets": [
        {
          "type": "retweeted",
          "id": "1892024895585804295"
        }
      ]
    },
    "TweetXMeta": {
      "newest_id": "1892040265302528116",
      "oldest_id": "1892039425959346193",
      "result_count": 10
    },
    "ThreadCursor": null,
    "Error": null
  },
  {
    "Tweet": null,
    "TweetXData": {
      "author_id": "489069983",
      "entities": {
        "urls": null,
        "annotations": [
          {
            "start": 72,
            "end": 79,
            "probability": 0.6225,
            "type": "Person",
            "normalized_text": "FAYEYOKO"
          },
          {
            "start": 102,
            "end": 114,
            "probability": 0.5536,
            "type": "Other",
            "normalized_text": "PromiseToStay"
          }
        ]
      },
      "id": "1892039425959346193",
      "possibly_sensitive": false,
      "reply_settings": "everyone",
      "conversation_id": "1892039425959346193",
      "public_metrics": {
        "retweet_count": 120,
        "reply_count": 0,
        "like_count": 0,
        "quote_count": 0,
        "bookmark_count": 0,
        "impression_count": 0
      },
      "edit_controls": {
        "edits_remaining": 5,
        "is_edit_eligible": false,
        "editable_until": "2025-02-19T03:32:06Z"
      },
      "text": "RT @chakrulo321: Who is here for trending? RT and Reply \"ALWAYS HER FOR FAYEYOKO\"\n\nFY NO MATTER WHAT\n#PromiseToStay",
      "context_annotations": null,
      "created_at": "2025-02-19T02:32:06Z",
      "display_text_range": [
        0,
        115
      ],
      "lang": "en",
      "edit_history_tweet_ids": [
        "1892039425959346193"
      ],
      "referenced_tweets": [
        {
          "type": "retweeted",
          "id": "1892024895585804295"
        }
      ]
    },
    "TweetXMeta": {
      "newest_id": "1892040265302528116",
      "oldest_id": "1892039425959346193",
      "result_count": 10
    },
    "ThreadCursor": null,
    "Error": null
  }
]

@alvin-reyes alvin-reyes self-assigned this Feb 19, 2025
@alvin-reyes alvin-reyes requested review from mcamou and mudler February 19, 2025 16:10
mcamou
mcamou approved these changes Feb 19, 2025
View reviewed changes
Copy link
Contributor

@mcamou mcamou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

mcamou
mcamou approved these changes Feb 19, 2025 •
edited
Loading
View reviewed changes
@alvin-reyes alvin-reyes marked this pull request as ready for review February 19, 2025 22:11
@alvin-reyes alvin-reyes merged commit 79150b0 into main Feb 19, 2025
2 checks passed
teslashibe pushed a commit that referenced this pull request Jul 14, 2025
@alvin-reyes
feat: extend the query search to get more details (#26)
300f9cd 
* fix/result-chars

* get extended search data from query

* get extended search data from query

* introduce twitter x data from x api

* add the count log

* remove duplicate field

* add meta

* add meta

* add meta

* add meta

* add meta

* clean up
grantdfoster added a commit that referenced this pull request Oct 13, 2025
@grantdfoster
fix: twitter-guard-and-job-definitions (#26)
deaef62 
* chore: add job arguments and job struct to types to prep for acceptance tests not needing ego imports

* chore: add count guard on twitter args

* chore: adds guards

* chore: adds var for max results

* fix: error message

* fix: allow unmarshalling on job types

* chore: adds unmarshal to job args
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mcamou mcamou mcamou approved these changes

@mudler mudler Awaiting requested review from mudler

Assignees

@alvin-reyes alvin-reyes

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@alvin-reyes @mcamou