Skip to content

Conversation

alvin-reyes
Copy link
Contributor

@alvin-reyes alvin-reyes commented Feb 17, 2025

Changes

  • additional logging on the twitterx scraper
  • fixed the body marshalling from the response

Verification

Run a job

Response body:  {"data":[{"text":"@BikiniPenguins Nice One1✨","edit_history_tweet_ids":["1891498582269698308"],"id":"1891498582269698308"},{"text":"@L_No_One1 Happy Birthday Dude i love your content have a nice day Marc ;)","edit_history_tweet_ids":["1891156995798954439"],"id":"1891156995798954439"},{"text":"Thnx for drop #DeepSeek Nice opportunity! \uD83E\uDE82  @omeiza_1 @SchmidtFras @ChipPin1011 @fdhlftr @CathySwarsky @bohdan4M @realJoelToni @alpha210x @Fmcd2121 @illuminate_one1 @astery404 @quizzerrrrr @fastRCNN https://t.co/jc4PSQWUf7","edit_history_tweet_ids":["1890460015934234667"],"id":"1890460015934234667"},{"text":"Thnx for drop #DeepSeek Nice opportunity! \uD83E\uDE82  @omeiza_1 @SchmidtFras @ChipPin1011 @fdhlftr @CathySwarsky @bohdan4M @realJoelToni @alpha210x @Fmcd2121 @illuminate_one1 @astery404 @quizzerrrrr @fastRCNN https://t.co/tvbaf86rpT","edit_history_tweet_ids":["1890239694505726032"],"id":"1890239694505726032"},{"text":"@Ge_One1 @_DSKDAO_ @ALLDAYDSK @4FishOutOfWater @LongedEth Nice\uD83D\uDD25","edit_history_tweet_ids":["1890150779786248229"],"id":"1890150779786248229"},{"text":"@Ge_One1 @_DSKDAO_ Nice.\uD83D\uDD25","edit_history_tweet_ids":["1889200793657483673"],"id":"1889200793657483673"},{"text":"im eligible nice\nBGMjKE @okaymizzy_ @NoGr1sha @DoTerreyo996 @Abe_One1 https://t.co/ilwJywPUId","edit_history_tweet_ids":["1889133671480725584"],"id":"1889133671480725584"}],"meta":{"newest_id":"1891498582269698308","oldest_id":"1889133671480725584","result_count":7}}

Verify result

c0c01e6c-1bd8-4bb3-9d60-41778fe1f240

AAIAAAQAAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAsAAAAAAAD/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAArkGif3X56I4TI3GzFEd5pFQCIoF8Nhl5LE7QYekiu+IuPr2UhBWSRtkbKKDYB/fLEIfuvyVdGDpOX1sOa1iPjF2e46ul7bv/JxGVU/JBuOLki1vdpmXee36UJCf1iRrEb271o9EegF+8DKgaXRLXPy/svdMh3Axw5AABa7/iCzmDodWjs52DlLHJ3khkyklIpQuIGqztnDP1v22GB/eTPhU3u3oocmDVJQY6VfdhUNHBSQQyO+RpI4+UZLGjW6bXhAKBsR0iiILKIm/8vN4nhDUs2+hbqDp3LGLQjh2YDo6DPpgJa6MD7dNkSeCLsyLAe9BQN2Yp+u5qGI82gT+o42fRm5qXF2Vura9W/o3H+8rZjg8Ijr+bIKKgu2OgjwrOtet2YZWfuedhLf4cmyDIiatrbhESnv1TyAP5WB81cyWIb8htIR7Hl/kVqnst+VXZdjppHTIPytOZDVm/pxaR8yxqLZJgLDvghJOH+iycXVF1Vld7aLiSWp6L6Knd07jsKy7yIyUeh7bm3dR/xVn75rxMUyPr+AOkxl352LEkyIkODTdOfuMBBbAYZDTQUZXMN0b4IbgAMsp62b0jLJBDQMf8eepm33H/8RdEI+7ztgD/Y2PPMivegjvHNm4ZW37pXXv/yNiCSC27gxJ6U2qbTKPseaLom81Kq5QMpBKmiK2oOnXB9Tzdvb214a9dP1nDmxNXBoORKhTX8eANsdYzWAAa1FhMeOjbNE6ai2SmqEBTq80gm2QoBi0QQLJ36toJ29D93tTGetYN1YTbVSLvAK0T+2wCMNxNr77izmZVqZ5/qSTqXykXFY3M7B4Y/FZ9Z4XEUkT6KNafrADqq92VOLjp5f43/V5wqLtQO0HMwOKT1yH5+q0G04nEjtn1nBztgIaUc7XIRyjRIBBQF56fXkAxwo+2bm1HPKbAAvRVNZCsTfOBIeQ/su+TeMtEzoq4v3yJ9WaGhTge/kMeIPEvpl3boafgJ5nlxaqpHSUH9NKVK4LXKUzsGTy++UA+V10AOVxX209mvmBXQk1ohEnKz7i3CR55CrbjJx0MsHH2BzP2qLDIprYy+2Bu8ygZVjVNG9UdpzGaSKV6B9EF74fb8AjSHKwf2oVw5QHb8Nv7Rio8ZMgYljJs5GOQ31Mvw4w9h7C7vFUqsIKbFTO0aOFtWXwfXk/AYl9fn/DO1AhZCSJL2Cjk/FMmr5+OYGVyPnGYGaotzM55D51OrepcAfFhwVm4bhgHosI2mv8pGSEw7/tsZCz2a545Zesp/iSxydNUooZqS07RmdTsqjcmwFl/u2y8Iq4QjSpiH1C3XhqIjIY99Ex00tNmS+4Pmp9v5cMDMHuN2rkrCFS8Jw30Cy4sYYVE1yWbUKs9ptwAd63STtB6G7+61YI3pTRRjbAYMYRVYHQPsMS8Bk6AzePG2TQUJdtG01PQWa00UWYiFn6+ChRUVWlpuupbLlR/ZAMsJybe2wUO30HFPr+ao0r8f9GxLGUa7brjuKC1ocZ3yNNxKjz1OVc5uTOocO4c6+rGCVuJZDJhLz0N1hYfcqH0VEh6l17Mzu6++TNlCZjM2c+sqhW2WNDGbajBwmLo2/b5ppNGYIIzX9KiTciifm0zjM4ZGG3WRpPmGef9y+0IDgXeV48JjTPP4skFSdzUxi36kOqv+evAeUWuI/MwVW91MpgvKcEqECQ1A6PwEkk1lTgpV31RxZzukSWyWgmuqSY8m7pKSaxlUvyBLID4IZyTw99tYbTh+pIHOvNOFGM079mWzVcUD6/vcfolleXf5+ONYesH/BBD3cyCFoEnEa+q6zcP2mMDj2Z2t8vJGyyEwo2Sbyc9n6uUBieRTXKFisT3VvmGcY7SE1EVnResT+PNDXATM+itPKIxG1Bfnr5x50x/zb5UDfnVrbXHRxu5rYBl/46HghOVzBnammmE7zwv1guJoCnyPwOSK8nU67Y4YhOwDGzkbhLEc/krQCKetWRV9QT6o3BhGG5+H7d00GC/PcVSh/4MBtuhVI47Xdt9dy8k/4VAC61XSMiWxb4h2l11bipmRpq0rFTTPUJ+9hF4V1Gj2ZVe+CqxUNt8NrWabOaZh+M0Z0+ULMXdmY/BVNEwipFhQOusSxh3xTtiz5ty+U/qlXoEFsudbfzVY+Mnz1BV4I1SMuGoLptfCZ6ucyd5GuMkLI8rhvotrWjK+dJX85C4RPSKfFHrbSJS8JSLuwDvvJgWK/T8B31A7cqPuoFYDJ2/fGnASov1x4Av9u1zndH9rQPQiOnaUEXggNrg7U639U9xYjtwjH5WU3F/J0gKXJ/iGPuTnb7Z73cT3bj/DvJna7XiV/ZRaQP3d/aDsD4J/EhnndNk+flIP35N5rXBwn/B/X5cXbyO8oIGRbW+yNlUxffJCBXxXz0BqklsgLZ4ERnwW2Gl8wLUW60ubZQ28a9klFNLmsAD8P6vKHNfEzbEuT/wHDT57GDrmlypP892LvZqiucs8vzCH5cVKkvpc6aWKET+P1BqEXeOxsLSkdMdPfqP4KxzrocXE+8Khuf2ZWy6PLzXSWshdJgKa95AF0oZckpha4gakreL51ierw9hethBZLpVYJ91qrJJt/T22rNNUf7yoPQUYb5akNwn6WfnV7Z7dH5z8Yfl6V1FFn28X+tOesT8ElIsGiC0Vyl446xheBmhA6xiOEmGOiCDLyxdKYoa4TASlo+TY4SDJqhjmRyAPLIeKEN+itXyCgPQ6zKNytFpqI3tS4jMG1uSreZDSOig9Zl4BA7APiULfPv3pRu1MxPa0Zu7D+EhMfHArrkuiEsQtuIfN1kgo7/mzmk5Kidwk0IXmfoxQJDM2I/3kJJl3zEj/HyQWOGVTOoXaYD9Ts0vHLHk29sRw7lt+f3BdZ8SwOBV3q45nxOWtSuk6nbnXDhf0jKJK4rLCGnZq2XWm3A8lT+qU4VGgnAfMMu5wiM8VgFzVTzxTap78dAFpspPI02hp+7iJaZQpjtpF0kFrAh2I7mfnwZbQA03cQLOlNqJK9D1gYRrfYCydkp9dXCCFdeBBrkiWzss1J31JQUPhobER9tsf7lGwUeD/UZiHE8aHi1q0LBzC9tiRFfBuHz31tJNKPbMA6UvcCBpBo9Qma5Ymw/0cp6q6ZrI3OJjVx6Yz4yZoO0ALEGy3qOgWvdCdkO2Plo3a06a0GKNfcDRF1b/qxDdT7TbjKB7+WpBJVk+ePKElR5/ZZEIgRYFrCiVu7kf1fj2VfnYXXBkmqTJijeeJ++C+k/4eCOg9KgTR77/ksQoTs3zjqnd7YOBXQ7J+xVT+LQGCXR3u6F2/HfG46G2RXJhpEoXd6sKz7BZzrIYYcgynUPXdHOeyHzt55A5jR9kAPb1+fssaBVbMSw8aci94gY4JzXuj+kbdKBAn5AscFdF/I09kLcZUcVGnVbp3acvoijrsFtwgVXF5yvYXrc4GNrrY4hAUohS2dFZ9Vh9SYNhbqBH07H21gYsXS9Ain0YpNcWI4wsKuab3pjUGH8RT+UB/SxNjq7RJGE9/P70zOS6QFNMUSCWaU0caoNp5lOabD+dgaMkGQtm2DJpFBPa9y7gh4IFjGxn5YnxgfynNRk28YYGcaP9Np6L47tx5CNX/mB0dn5Q/adJfP9tCDnjN2LfEG3X25W2v1pbZBnOKN5cHS1TGadVcDyN8ZbugUewcMbWFfU+n6YZBSiDFc4JC/6CiAelPOqop93GHGlKk+wDT4yib9qFCngf9A3hMa55esfu3AK5jXf8owdSCDCrZMv6PPEkT1hjK08bquYanvHYqC74ZIHK2U9UyKg2ErXHHR6R8eF3XkPTtyoLQ98v4RPXJD2wNLg2zIkiQJFuR6ctZ2quT96Jp3I6NRbpKDNaa7rIvExhuxxq1g+NJKIBrg0qdF+hTdfMqkz8pYo/u1WUNP6pssuTrCeSx12IDWFSiZFmj/Zu1482wS9ZigKdjSL0qyOC1ATo65HFebxGncsrFJHC/MCCnyk/l97bIwTlYyThl0TiFJxqg93Me3qomcAnZF6ShQAqRgRV3r4f6sXtzG7FEDwfkcrEfJ6pCKFrsdEC6ptTIQbPXXdEBrJ9hWoS1Xt/+ng5kOeK6VlI8Oc+Ca6CF7d3a0TxSYSeTlxO6Z+bTAE7lTQTotvE5AHP+qXBnvk08dCB3lJRrmc13vCNmxJseTU0Uaz6o0NNjGQbpm48aXPUwagqhDrTGV6+QWr/ERenXtoU9SyjV7EJZMkGOO6sfSJGgmY31MCELZ4Lp9yFwPqb7gmjREMvK4wZHkb5mRRDxl6knik8cibCn7HDNha9+0TQ0pHl4p088uVMrNOSPqQtMszRcDdu8XZpY/v9oMnEpXaAdTMRklhO3TUPRPBc/KBPnqNrLypf+SOja9lSBBpkQ3DCGsthpv6qYUAWsNPYPUXmbSYjer0DCSHUoNzn0/hxfJG0CwyNZhsyNsvoaxUDzI2YSYimssHIfxJjSRxNa9G/ESZHMKrIklWNtN99WxnPsZ27uAbJ8SvUENFT7FBQIxKDx8tdVwWOBHMxTDxAm0WjRDYwXIuBADSFzde20iI+2Br3ACHE90nWf/U8oR7dZn+DKkhMCiuYMRf8S38OcdgSdoc3yAmEowQuGqwLCljTPc6zSVlQD9g7sq0jTXyZH0eDgP0tAxPoJluCa6o+va9GmK2tOJXKSpsW4n86TnzPAtfCBXc53kaGm0lgHZp6OxSiGBRiSTAsNtaC2bQGJRBX/NRTJY/975Pvt72HSfSkoTkSi4HnvZOCnMpypw4GFy4a3kCpzPKWm0T7UwIL688oFIxvr+5gODJVGf/FgzRTrKPRLsTACzhyCDw9EZvbMDdQi9H/71fqB8teQKlURjUrQzWJXLBLR3N3yWVDC46/HJBompDXJko6LUkmqOxSUTbdzalTQ731L8uqNsbLkj/cu/iWuv7ZNGIIFnWB4WZTUJ1TS3gnaMBPI7Q+epE3QDXdtKV01OhF/HPHQoBrNBLY128DBYAULGUde137ihqGlwPK/3ygIFZgPtbwE0DQSsV7kw8Sv10H7ehAAXMbe/d5CCwujSg01aruvoE9nf45094uklzapheSLnXCHICUdxN5qFoqhD5dUuzwp7/JFzF9AA0ncoGXT/obv+EeNa8MOeUVK0+vRacrV3+P12cWM7GDCNTusyzMFYA0NFD2ki+edimNVFl8KomFsQfvZXz5tr0rvhkQRwjAcoiUBTV8hR0w+HpFgp92x7q7lzed9tPUacWjHCQhmPcsc/odEay70I3k/5tLnd+TuB1RP0YGL95notHpvTnFNivWufPkPeb2+cTenKPC+msboy+vRWKjVYEVIJMp2xQLWOHlBPw0XIzqDIWpM+2XvsqodOTIEjXYYFvNi/cQxQKAmy/pdvUAgsENH6WAV35PWndsf5oP4PM0C1z1BlKFtPJJflgTGzG0Ac8GWb7oX2xp8KwdTA0mvdlsGIYjaww21uRJ7fi2Ang2TdmjIQjL6vNQOiXDtA5c71I6oEO/+gRkE/T98oVHYJBXKZqXk+sQCcW4Mh5A2qWC9QS/FkG6iUcsTrCSWlwH36Pzu50KfSPIaFVooiBJ1hgDgGbSSPJt8CFxniOMghdgg7EDQZbFdp6p7LwlwdbgodmXag/nvRzkAH2sn2fycJHIdLBUtrWLJSixzcIFh8kwPRCbiLkIxCJDteaax6WiKUYCKx51HnxVB1k58cN+iU8ACzE+PUedetLoZU5dyN+Vsb1Qw77+LEcP+uBMST2noXtI1V0MjMwWY6L72hpq2ebiaQ1Lzm1i9J7KOPsdJKoIuinSMky0l52b1lFSS0EnDSI36OuEQIIOR5ue7AYpDeWriIn9rpmpzkt+Rp5yNc0ArFqGclgCSQqBojJKVh8kVLYoLjQqXCleeImsNdl8rdjouSyVxO7OheTrMovuRZsRSP9m1HvXq8LP0uwDKgjKmIW3yaXDSaqoXGGcMY8ou+oI/uz9TOnBNP7gJd1vHuXJkRtZ5RR+te5UNO/t4PYbI8VOijIE9DKAY8nfrfLRt5lkmSZK7kZvOuU1AM1XRAaTlxMFOAMzwqBLRbNWlj8+kBYGvx7+L7/Lr1PsIs6yV8LEldpaLjMQnJANZvJAbZoIeldUdVecXFwh9XTk4WOqvEcqfzR2SvNnG3q8dah5NycJBaoYYR9n+97lFazykGo5YHFJ/vCJnodzLXCOE6Dj/n2eqXUXLnabTJQFfuKdnGqA3H1IdB7QK4Vyt3Rcr8LOqy3qq9SU/dir4awWaC8VXDoy+ld9F3y0ogJlorM3wVrZ/gs7Ud2Fy9N/ZF77cYeBasjCLrTeqPmM3ax/g4+fz1Ctcg963v0bUoRnmo1ZWzAwQ3YcQvsT1Yb/mak8vlhPK9bFUq+cGOrgMBDk2zhvvFaut92wvSmrAHedRLiX0WEhIW2kC0GsbE/59lF7Nyuq705TDUtQcJCZINpb2Gek1xEFdutcRv0jIYN6jdecEcUlG9VB+BF/9SeyXHwrXxMYWh3Hvo9slgVHOYKTxUkUY3jaL+L1Um7pCCfM+1awNghLWKGNEr3ANKBahxTm+cKVchBfCElCXcNFdknp6HWV39x7Hqtzbmm3WO7QXWQa0TIfuF6nLorsGYNeWi4kvReNF3d2W28yBFp3dqnH8IpZAWejCXXUXmWyoBU1P8XvYXc143yruP1dKf9L/cxcr859TuDU5u/VWFpmq+iKzzerojufiCyYfJzYftjb+AJ/9Nn3XV3a2toyLdme3ohu9Of3iVaTpLi

@alvin-reyes alvin-reyes self-assigned this Feb 17, 2025
@alvin-reyes alvin-reyes requested a review from mudler February 17, 2025 19:58
@alvin-reyes alvin-reyes marked this pull request as ready for review February 17, 2025 20:02
@mudler mudler merged commit a521e37 into main Feb 18, 2025
2 checks passed
teslashibe pushed a commit that referenced this pull request Jul 14, 2025
* use TwitterResult rather than interface

* move tweets above logic

* add logging on the twitter x client scraper

* add logs and search param move

* log response body before decoding

* log response body before decoding

* pass body to appropriate scrape

* pass body to appropriate scrape

* pass body to appropriate scrape

* properly decode response

* log the body

* escape characters

* log tweets on scrap

* clean up

* clean up the fmts

* lint/fmt error issues

* lint/fmt error issues does not support error-wrapping directive
grantdfoster added a commit that referenced this pull request Oct 13, 2025
* fix: add max pages to llm args

* chore: fix test

* fix: rename to items

* chore: fix test

* fix: rename vars

* chore: relevant llm args to uint instead of int

* chore: temperature as a float

* chore: fix test

* chore: fix llm test

* fix: llm test

* fix: temperature test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants