Check availability of profile pictures and fail-over option

[Orbiter]

This feature shall be implemented for the search page but shall also be applicable for all other places in loklak_webclient, where a profile image is needed.

We currently have the problem that some profile pictures are not loaded from Twitter in all cases. In case that a load from twitter is not possible, we need an alternative image. That image shall be provided by the loklak_server back-end.
The following process shall be implemented:

• instead of embedding a profile image directly from twitter, we first show a placeholder (I will provide that)
• for each placeholder, a javascript function shall try to load the image from twitter.
• if loading from twitter is successful, show that image instead of the placeholder
• if that function fails (i.e. time-out or error message from twitter) the function must try to load an alternative from the loklak_backend
• if the back-end returns an image, show that image instead of the placeholder.
• if the backend does not answer (time-out or http error status code) the function just terminates and does nothing (the default image stays in place) or if an alternative is returned, it is displayed

The request to the back-end must contain the screen_name of the user.

[Orbiter]

the placeholder images had been uploaded with commit 23a6223
Please use the correct one according to the actual profile image size that shall be displayed (currently only the _bigger version)

[Orbiter]

This feature will be supported by loklak_server issue loklak/loklak_server#31

[fuksi]

@Orbiter, this is now ready from the front end.
Currently, it will use the given img as the placeholder.
The current parameter for the request to the server is

...?q=screen_name:SCREENNAME

Please open this issue again when server is ready, and comment here if the query url structure changes.
Cheers!

[Orbiter]

Ok I will reopen this issue as soon as loklak/loklak_server#31 is implemented

[Orbiter]

The backend issue loklak/loklak_server#31 is implemented. Please use the following URL to access the cache on the server-side:

http://localhost:9000/api/proxy.png?screen_name=loklak_app&url=https://pbs.twimg.com/profile_images/577512240640733184/fizL4YIn_bigger.png

This means you must submit two parameters to the /api/proxy.png servlet:

• screen_name: the user name id
• url: the url that you want to load from twitter

[Orbiter]

The latest version of loklak_server can now attach images inside the user information encoded with base64. Therefore, the following task shall be done in front of the process as described in #59 (comment)

• each userobject now may contain a profile_imageproperty which contains a large base64 string with the user image. If this property exist, do not load any other image but compute the png from the base64 string.

[Orbiter]

the complete process now is:

• for each user object check if the field profile_image exist. If it exist, decode it with base64 and display it as png -> ready
• show the placeholder image
• a javascript function shall try to load the image from twitter
• if the image from twitter arrives, show it -> ready
• if the image does not arrive, call the loklak_server with /api/proxy.png?screen_name=<screen_name>&url=<url> and show the image which is returned -> ready
• if still no image is available from loklak_server, give up (at this time the placeholder should be there already)

[fuksi]

I did a quick test and the result is that, for user of image request from twitter failed, requests to loklak server will fail too.

Quick demonstration: here I have search listing accounts, so I'd know which to test.

http://gofullstack.me:3001/search?q=%23loklak%2B%2Faccounts&timezoneOffset=-180

Images those are unsuccessfully loaded from twitter, would deal with the same result from loklak:

• http://loklak.org/api/proxy.png?screen_name=aneeshd16&url=https://pbs.twimg.com/profile_images/599573466392731649/Ek30zZIu_bigger.jpg
• http://loklak.org/api/proxy.png?screen_name=devnewsbot&url=https://pbs.twimg.com/profile_images/484741856933060609/FTZs8ohE_bigger.png

Images those are successfuly loaded, can be requested again from loklak:

• http://loklak.org/api/proxy.png?screen_name=SpringValleyKep&url=https://pbs.twimg.com/profile_images/465550354218622976/0iWeRrJc_bigger.jpeg
• http://loklak.org/api/proxy.png?screen_name=SamiesHungry&url=https://pbs.twimg.com/profile_images/589832071129870336/rHQBrLAz_bigger.jpg

[aneeshd16]

@Orbiter , @fuksi this can be a possible solution: http://avatars.io/
In short, this service claims that they will always return the correct URL of the profile image.
Eg: http://avatars.io/twitter/aneeshd16?size=large leads to http://pbs.twimg.com/profile_images/610447150422560768/m777s_is_bigger.jpg
Simply remove '_bigger' in the URL to get the full size image.

Edit: The service is in beta, and it is an external dependency. We don't know if they will be present or change in the future. But for now, it works.

[aneeshd16]

Additionally this page tells us why we are getting these errors:
https://dev.twitter.com/overview/general/user-profile-images-and-banners
Scroll down to Outdated profile images. The scraper stored the url of the user when it scraped the tweet. However we can easily get the newest profile url by using https://dev.twitter.com/rest/reference/get/users/show since we now also have auth tokens.

Possible solution:

Whenever the image is not received, call the twitter api to get the latest url. This call can be made through:

• loklak_server: It will be able to overwrite the previous incorrect value. Then it sends the correct url to the client.
• loklak_client: This will be faster, will show the correct url immediately. After that, it can send the correct url to loklak_server so that the backend can update its data.

[sudheesh001]

Great find @aneeshd16 But I think its better if the loklak_server does this than the client, that way we know that the server always has the updated images, Maybe it could be a cron service which runs regularly to make these updates. What do you think @fuksi @Orbiter @prasht63 ?

[Orbiter]

a cronjob or similar is out of question. We already have over 8 million users in our database and there is no point checking them all the time.
What we can do: use the proposed update process in case that the /api/proxy is unable to retrieve the image. That would mean: no changes on your side.

[mariobehling]

@aneeshd16 Yes, great find, but this app is in Beta and we probably have to pay later. I do not think it is magic to get avatars from Twitter and if that does not work from Gravatar. Supporting Gravatar should be a feature for the future anyways. Not sure, if that is high on the list right now. What we need is a working prototype to get users. So, the current approach seems sufficient.

[mariobehling]

As the implementation here is ongoing, I do not understand why this issue is closed. I am reopening it and assign it to M2. If this issue is supposed to be closed, please explain why and create a new issue for a specific feature instead of continuing on a closed issue.

[mariobehling]

Oh sorry. I think I got confused by the label "closed" at some point. I see that this issue is open, but it was not in M2 yet. Cheers.

[Orbiter]

The image will be provided by the loklak_server using @aneeshd16 idea to load the latest url from twitter directly. This requires that a twitter api library must be included in the loklak_server (shall be done anyway) to load that information. The issue is tracked with loklak/loklak_server#51

While this is implemented, this feature is in hold-on state. If the proxy provides this function, it can be re-tested and if working this feature can be closed without any more coding on client side.

[fuksi]

This is now moved to milestone 3 until further support.

[Orbiter]

This is now moved to milestone 3 until further support.

The supporting feature has been implemented with loklak/loklak_server#51

I wonder why this feature has not been used so far, because it should have helped in the most cases already. I can see in the log that the proxy-servlet had not been called once, ever!

Here is an example call:
http://localhost:9000/api/proxy.png?screen_name=loklak_app&url=https://pbs.twimg.com/profile_images/577512240640733184/fizL4YIn_bigger.png

If the image url is unknown or has changed, the proxy servlet will find it out itself.
I.e. http://localhost:9000/api/proxy.png?screen_name=loklak_app&url=https://pbs.twimg.com/profile_images/i_dont_know_the_current_image_name.png will also return the image, even if the given url is completely wrong. The server takes the latest information from twitter itself if the screen name is known.

[mariobehling]

@Orbiter M3 is closed. So, I moved it to M3.5.

[sudheesh001]

I think this is already implemented, any images which fail to load, use the image at
http://test.loklak.net/images/anon_200x200.png as a failover.

@fuksi Please confirm.

[fuksi]

This is already implemented with user.json servlet. It's working alright. After a search, If img src failed. A request is made to user servlet to get a working url. I believe this servlet is getting data from twitter as well :)