Gallery fetches metadata for 800+ pictures, consuming a lot of data and causing flickering

[nicolas-raoul]

I noticed that the app keeps making tons of requests (several per second) to get metadata of some of my very old pictures, taken several years ago. I would have to scroll down 800 pictures to reach these pictures.
Logcat:

10-11 23:35:20.896 29773 30101 I okhttp.OkHttpClient: --> GET https://commons.wikimedia.org/w/api.php?action=query&format=json&formatversion=2&generator=allimages&gaisort=timestamp&gaidir=older&prop=imageinfo&iiprop=url|extmetadata&iiurlwidth=640&iiextmetadatafilter=DateTime|Categories|GPSLatitude|GPSLongitude|ImageDescription|DateTimeOriginal|Artist|LicenseShortName|LicenseUrl&gaiuser=Syced&gailimit=10&gaicontinue=20190403071047|Zabbix_4.0_web_UI.png&continue=gaicontinue||
10-11 23:35:20.902 29773 30101 D CookieManager: Domain:commons.wikimedia.org
10-11 23:35:20.923 29773 29773 W StaticLayout: maxLineHeight should not be -1.  maxLines:2 lineCount:2
10-11 23:35:20.924 29773 29773 W StaticLayout: maxLineHeight should not be -1.  maxLines:2 lineCount:2
10-11 23:35:21.039 29773 30089 I okhttp.OkHttpClient: --> GET https://commons.wikimedia.org/w/api.php?action=query&format=json&formatversion=2&generator=allimages&gaisort=timestamp&gaidir=older&prop=imageinfo&iiprop=url|extmetadata&iiurlwidth=640&iiextmetadatafilter=DateTime|Categories|GPSLatitude|GPSLongitude|ImageDescription|DateTimeOriginal|Artist|LicenseShortName|LicenseUrl&gaiuser=Syced&gailimit=10&gaicontinue=20190403071047|Zabbix_4.0_web_UI.png&continue=gaicontinue||
10-11 23:35:21.051 29773 30089 D CookieManager: Domain:commons.wikimedia.org
10-11 23:35:21.052 29773 29773 W StaticLayout: maxLineHeight should not be -1.  maxLines:2 lineCount:2
10-11 23:35:21.053 29773 29773 W StaticLayout: maxLineHeight should not be -1.  maxLines:2 lineCount:2
10-11 23:35:21.089 29773 30056 I okhttp.OkHttpClient: <-- 200 https://commons.wikimedia.org/w/api.php?format=json&action=wbgetentities&props=labels&languagefallback=1&ids=M77643295%7CM77643452%7CM77647672%7CM77701972%7CM77702248%7CM77725997%7CM77744936%7CM77745111%7CM77745158%7CM77746574 (1304ms, unknown-length body)
10-11 23:35:21.094 29773 30072 I okhttp.OkHttpClient: <-- 200 https://commons.wikimedia.org/w/api.php?action=query&format=json&formatversion=2&generator=allimages&gaisort=timestamp&gaidir=older&prop=imageinfo&iiprop=url|extmetadata&iiurlwidth=640&iiextmetadatafilter=DateTime|Categories|GPSLatitude|GPSLongitude|ImageDescription|DateTimeOriginal|Artist|LicenseShortName|LicenseUrl&gaiuser=Syced&gailimit=10&gaicontinue=20190403071047|Zabbix_4.0_web_UI.png&continue=gaicontinue|| (2426ms, unknown-length body)
10-11 23:35:21.103 29773 30072 I okhttp.OkHttpClient: --> GET https://commons.wikimedia.org/w/api.php?format=json&action=wbgetentities&props=labels&languagefallback=1&ids=M77643295%7CM77643452%7CM77647672%7CM77701972%7CM77702248%7CM77725997%7CM77744936%7CM77745111%7CM77745158%7CM77746574
10-11 23:35:21.105 29773 30072 D CookieManager: Domain:commons.wikimedia.org
10-11 23:35:21.151 29773 30125 I okhttp.OkHttpClient: <-- 200 https://commons.wikimedia.org/w/api.php?action=query&format=json&formatversion=2&generator=allimages&gaisort=timestamp&gaidir=older&prop=imageinfo&iiprop=url|extmetadata&iiurlwidth=640&iiextmetadatafilter=DateTime|Categories|GPSLatitude|GPSLongitude|ImageDescription|DateTimeOriginal|Artist|LicenseShortName|LicenseUrl&gaiuser=Syced&gailimit=10&gaicontinue=20190403071047|Zabbix_4.0_web_UI.png&continue=gaicontinue|| (1494ms, unknown-length body)
10-11 23:35:21.158 29773 30125 I okhttp.OkHttpClient: --> GET https://commons.wikimedia.org/w/api.php?format=json&action=wbgetentities&props=labels&languagefallback=1&ids=M77643295%7CM77643452%7CM77647672%7CM77701972%7CM77702248%7CM77725997%7CM77744936%7CM77745111%7CM77745158%7CM77746574
10-11 23:35:21.160 29773 30125 D CookieManager: Domain:commons.wikimedia.org
10-11 23:35:21.163 29773 30067 I okhttp.OkHttpClient: --> GET https://commons.wikimedia.org/w/api.php?action=query&format=json&formatversion=2&generator=allimages&gaisort=timestamp&gaidir=older&prop=imageinfo&iiprop=url|extmetadata&iiurlwidth=640&iiextmetadatafilter=DateTime|Categories|GPSLatitude|GPSLongitude|ImageDescription|DateTimeOriginal|Artist|LicenseShortName|LicenseUrl&gaiuser=Syced&gailimit=10&gaicontinue=20190329013233|Purple_flowers_forming_in_Azabu.jpg&continue=gaicontinue||
10-11 23:35:21.165 29773 30067 D CookieManager: Domain:commons.wikimedia.org
10-11 23:35:21.186 29773 29773 W StaticLayout: maxLineHeight should not be -1.  maxLines:2 lineCount:2
10-11 23:35:21.187 29773 29773 W StaticLayout: maxLineHeight should not be -1.  maxLines:2 lineCount:2
10-11 23:35:21.196 29773 30055 I okhttp.OkHttpClient: <-- 200 https://commons.wikimedia.org/w/api.php?action=query&format=json&formatversion=2&generator=allimages&gaisort=timestamp&gaidir=older&prop=imageinfo&iiprop=url|extmetadata&iiurlwidth=640&iiextmetadatafilter=DateTime|Categories|GPSLatitude|GPSLongitude|ImageDescription|DateTimeOriginal|Artist|LicenseShortName|LicenseUrl&gaiuser=Syced&gailimit=10&gaicontinue=20190403071047|Zabbix_4.0_web_UI.png&continue=gaicontinue|| (2638ms, unknown-length body)
10-11 23:35:21.205 29773 30055 I okhttp.OkHttpClient: --> GET https://commons.wikimedia.org/w/api.php?format=json&action=wbgetentities&props=labels&languagefallback=1&ids=M77643295%7CM77643452%7CM77647672%7CM77701972%7CM77702248%7CM77725997%7CM77744936%7CM77745111%7CM77745158%7CM77746574
10-11 23:35:21.208 29773 30055 D CookieManager: Domain:commons.wikimedia.org
10-11 23:35:21.222 29773 30074 I okhttp.OkHttpClient: <-- 200 https://commons.wikimedia.org/w/api.php?format=json&action=wbgetentities&props=labels&languagefallback=1&ids=M77746573%7CM77746575%7CM77746577%7CM77746578%7CM77765278%7CM77790559%7CM77790629%7CM77857454%7CM77857640%7CM77857737 (1456ms, unknown-length body)
10-11 23:35:21.281 29773 30077 I okhttp.OkHttpClient: --> GET https://commons.wikimedia.org/w/api.php?action=query&format=json&formatversion=2&generator=allimages&gaisort=timestamp&gaidir=older&prop=imageinfo&iiprop=url|extmetadata&iiurlwidth=640&iiextmetadatafilter=DateTime|Categories|GPSLatitude|GPSLongitude|ImageDescription|DateTimeOriginal|Artist|LicenseShortName|LicenseUrl&gaiuser=Syced&gailimit=10&gaicontinue=20190329013233|Purple_flowers_forming_in_Azabu.jpg&continue=gaicontinue||
10-11 23:35:21.285 29773 30077 D CookieManager: Domain:commons.wikimedia.org
10-11 23:35:21.301 29773 29773 W StaticLayout: maxLineHeight should not be -1.  maxLines:2 lineCount:2
10-11 23:35:21.301 29773 29773 W StaticLayout: maxLineHeight should not be -1.  maxLines:2 lineCount:2
10-11 23:35:21.332 29773 30122 I okhttp.OkHttpClient: <-- 200 https://commons.wikimedia.org/w/api.php?format=json&action=wbgetentities&props=labels&languagefallback=1&ids=M77746573%7CM77746575%7CM77746577%7CM77746578%7CM77765278%7CM77790559%7CM77790629%7CM77857454%7CM77857640%7CM77857737 (1491ms, unknown-length body)
10-11 23:35:21.393 29773 30070 I okhttp.OkHttpClient: <-- 200 https://commons.wikimedia.org/w/api.php?format=json&action=wbgetentities&props=labels&languagefallback=1&ids=M77746573%7CM77746575%7CM77746577%7CM77746578%7CM77765278%7CM77790559%7CM77790629%7CM77857454%7CM77857640%7CM77857737 (1491ms, unknown-length body)
10-11 23:35:21.414 29773 30066 I okhttp.OkHttpClient: --> GET https://commons.wikimedia.org/w/api.php?action=query&format=json&formatversion=2&generator=allimages&gaisort=timestamp&gaidir=older&prop=imageinfo&iiprop=url|extmetadata&iiurlwidth=640&iiextmetadatafilter=DateTime|Categories|GPSLatitude|GPSLongitude|ImageDescription|DateTimeOriginal|Artist|LicenseShortName|LicenseUrl&gaiuser=Syced&gailimit=10&gaicontinue=20190329013233|Purple_flowers_forming_in_Azabu.jpg&continue=gaicontinue||
10-11 23:35:21.416 29773 30066 D CookieManager: Domain:commons.wikimedia.org
10-11 23:35:21.437 29773 29773 W StaticLayout: maxLineHeight should not be -1.  maxLines:2 lineCount:2
10-11 23:35:21.438 29773 29773 W StaticLayout: maxLineHeight should not be -1.  maxLines:2 lineCount:2
10-11 23:35:21.494 29773 30069 I okhttp.OkHttpClient: <-- 200 https://commons.wikimedia.org/w/api.php?action=query&format=json&formatversion=2&generator=allimages&gaisort=timestamp&gaidir=older&prop=imageinfo&iiprop=url|extmetadata&iiurlwidth=640&iiextmetadatafilter=DateTime|Categories|GPSLatitude|GPSLongitude|ImageDescription|DateTimeOriginal|Artist|LicenseShortName|LicenseUrl&gaiuser=Syced&gailimit=10&gaicontinue=20190403071047|Zabbix_4.0_web_UI.png&continue=gaicontinue|| (1623ms, unknown-length body)
10-11 23:35:21.502 29773 30069 I okhttp.OkHttpClient: --> GET https://commons.wikimedia.org/w/api.php?format=json&action=wbgetentities&props=labels&languagefallback=1&ids=M77643295%7CM77643452%7CM77647672%7CM77701972%7CM77702248%7CM77725997%7CM77744936%7CM77745111%7CM77745158%7CM77746574
10-11 23:35:21.503 29773 30069 D CookieManager: Domain:commons.wikimedia.org
10-11 23:35:21.546 29773 30127 I okhttp.OkHttpClient: --> GET https://commons.wikimedia.org/w/api.php?action=query&format=json&formatversion=2&generator=allimages&gaisort=timestamp&gaidir=older&prop=imageinfo&iiprop=url|extmetadata&iiurlwidth=640&iiextmetadatafilter=DateTime|Categories|GPSLatitude|GPSLongitude|ImageDescription|DateTimeOriginal|Artist|LicenseShortName|LicenseUrl&gaiuser=Syced&gailimit=10&gaicontinue=20190329013233|Purple_flowers_forming_in_Azabu.jpg&continue=gaicontinue||

I noticed it on the branch for #3916 but it might be happening in master too.

The app should only fetch the metadata of the pictures that are likely to be shown soon, maybe it can pre-fetch a dozen or so, but pre-fetching 800 is far too many, and takes a toll on the user's data.

[borisenkotim]

Hi, I would like to give this issue a try.
Does this occur for you right after you upload the image or at what point of the process?

[nicolas-raoul]

Thanks for asking, very good question, I investigated and actually the answer surprised me.
This has nothing to do with uploading, apparently.
It happens in the gallery (the main activity of the app, which shows your uploaded pictures).
The app seems to load metadata for all pictures I have ever uploaded, which is thousands.
When it has finished (it can take hours), the behavior is not observed anymore. Clearing the app's data and logging in again makes the behavior start again.
It will be easier to debug if you have uploaded many pictures on Commons. Here are some pictures to upload if you need: https://drive.google.com/drive/folders/1T5wfM16qb78XCvnf1Zunww4neg8BXJj6?usp=sharing

[nicolas-raoul]

I noticed that this fetching behavior also makes the home activity "flicker", the activity shows a brief white flash after each block of lines appears in logcat.

I made a video showing the app and logcat, one hour after opening the app. In this video you can see the app busy downloading metadata for a picture I uploaded 7 years ago, and which would take me an hour to scroll down to in the app.

[borisenkotim]

When it comes to the flickering, it appears to be a common thing and has to do with the asynchronous API calls going on in the background. The master branch also has this issue. A common fix is to use ContentLoadingProgress bar: https://developer.android.com/reference/androidx/core/widget/ContentLoadingProgressBar
Which is used to prevent this flicker. This would be an addition of a UI element and probably be an enhancement issue as a result. As for the excess metadata being fetched, I was able to decrease it by decreasing the Prefetch distance and page size and this decreased the amount of data being prefetched.

[nicolas-raoul]

@borisenkotim I unassign for now, but if you are you still working on this, please let us know. If no answer, someone else may be assigned to it. Thanks a lot. :-)