Twitter not working

[bradvatne]

The sample twitter link: https://unavatar.now.sh/twitter/kikobeats doesn't work, nor do any requests I make!

EDIT: Seems like 'got' is getting a 400 error from Twitter, maybe they are blocking this type of request? I tried using 'request' package to scrape the HTML but it returns an incompatible browser page.

I've just decided to sign up for the Twitter API and create my own script to fetch the profile images. It's rate limited at 900 rq/15min so I don't know if that's a viable solution for this project.

[Bobface]

It seems Twitter changed a lot of stuff today. The scraping code is not working anymore.

[sserghei]

@Kikobeats seems that fix doesn't work(
or may be the last version was not deployed yet?

[Kikobeats]

Looks like a problem related to Twitter detecting requests are coming from production servers and they respond with different HTML markup, being not possible get the avatar URL.

That could be resolved to perform the Twitter request through a proxy server. Perhaps that means we need to set up a proxy server or using enough good free service (not sure about that)

In the middle time, you can get Twitter avatar using Microlink API

https://api.microlink.io/?url=https://twitter.com/kikobeats&embed=image.url

[ahmadnassri]

if it helps, I've been using Luminati's Super Proxy successfully for months...

[d-ivashchuk]

Hey maintainers! I am curious if you are working on some solution to fix the last changes by twitter or you recommend switching to some other services/fetching images natively via twitter api?

Thanks a lot for your work!

[Kikobeats]

@d-ivashchuk if you self-hosted your own version (e.g., using Heroku) it should be working since the current problem is because the version is hosted on Vercel (AWS) and they detect the IP address is coming from their servers, serving different HTML markup, being not possible get the image because is not present

[Kikobeats]

Likewise, I need to investigate a different hosting alternative; suggestions (and sponsorship) are very welcome

[frankdilo]

Please be aware that Heroku is deployed to AWS as well.

[d-ivashchuk]

I am curios if it's possible to self host it using docker?

[Kikobeats]

hum, what about https://workers.cloudflare.com?

[Bobface]

@d-ivashchuk Yes it is, I am currently using it that way. Here's a simple Dockerfile to make an image (assuming the repo is cloned to ./unavatar from the Dockerfile's perspective):

FROM node:buster

COPY ./unavatar /unavatar

WORKDIR /unavatar
RUN npm install
CMD ["node", "src/server.js"]

The application listens on port 3000.

[d-ivashchuk]

@Bobface is it working now run in docker? Where are you running a machine?

[Bobface]

@d-ivashchuk Yes it is working. I am running a machine with one of the more "old school" server providers (dedicated, year contract, etc.). I had this server already before these problems occured though.

[maxzzze]

Deployed with docker image on GCP and Twitter still does not work.

[Booligoosh]

A potential solution could be to use DuckDuckGo's Twitter API proxy, eg:
https://duckduckgo.com/tw.js?user=kikobeats
It contains the avatar URL and then unavatar could just swap out "_normal" for "_400x400" like in the existing getAvatarUrl function.

[Booligoosh]

Alright, I've just submitted a pull request that does that! @Kikobeats let me know what you think.

[ghost]

A potential solution could be to use DuckDuckGo's Twitter API proxy, eg:
https://duckduckgo.com/tw.js?user=kikobeats
It contains the avatar URL and then unavatar could just swap out "_normal" for "_400x400" like in the existing getAvatarUrl function.

Not working for me, it just returns:

{"user":"kikobeats","description":"","profile_image":"","location":"","current_status":null,"status":"","friends_count":"","entities":null,"name":"","followers_count":""}

[Booligoosh]

Ah true, I just tested it behind a VPN and it seems they don't accept requests from IP addresses of common VPNs, Vercel, etc. Disregard the pull request then, sorry!

[ghost]

It also wasn't working from my home connection, but interestingly Ctrl+Shift+R makes it work.

[Booligoosh]

Actually, that's really weird, it seems to be working all of a sudden. Maybe it's due to a temporary IP block or something? Demo:
https://unavatar-twitter-test.vercel.app/twitter/kikobeats
(I'll probably take the instance down in a few days so don't rely on it, it's just a demo of the code on the fix-twitter branch in my fork).

[karlhorky]

Something that works up until your limits on Cloudinary is their Social Media Profile Pictures API:

https://cloudinary.com/documentation/social_media_profile_pictures

Just switch out the demo in their example URL with your own account's cloud_name (and the Twitter username with one that you want to fetch):

https://res.cloudinary.com/demo/image/twitter_name/BillClinton.jpg

[driesvints]

Thanks @Kikobeats for fixing! 👏

[Kikobeats]

very low quality for the moment but let me figure out if I can improve it 🙂

[Kikobeats]

For people reading this issue: high res Twitter avatar resolution is back!

https://unavatar.io/twitter/kikobeats

(also with a new fresh unavatar.io domain)

It's using Microlink under the hood since the service has the prerendering capabilities for getting the right avatar URL.

Microlink API has some free quota, but in case you have a high demand for this, consider buy a pro plan and just pass the API key as x-api-key request header.

[karlhorky]

Hm, I guess I shouldn't be using this for awesome-speakers, seems to break a lot of the time:

[alvarotrigo]

Hm, I guess I shouldn't be using this for awesome-speakers, seems to break a lot of the time:

Yeap, same results.
Some avatars won't show up.

[Kikobeats]

The Twitter integration is getting hard due a headless browser is used for getting the avatar, and the Twitter site perf is terrible these days.

When an avatar is not resolved is because Twitter timed out, working on it to find a sweet spot, that also implies upgrade the unavatar.io infrastructure to have better vCPUs.