New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up Amazon app #457

Closed
Gitoffthelawn opened this Issue Aug 5, 2016 · 48 comments

Comments

3 participants
@Gitoffthelawn

Gitoffthelawn commented Aug 5, 2016

When the Amazon app (or the Amazon Underground app), which include their own app store, fail to install an app, the downloaded app becomes an orphan.

This often happens when the user tries to install an app from Amazon's app store, but the device does not have sufficient available storage to perform that operation.

Also, I wonder if there are more opportunities to clean up waste from the Amazon apps. I've seen devices on which they are occupying hundreds of megabytes.

@d4rken

This comment has been minimized.

Owner

d4rken commented Aug 6, 2016

Depending on file locations this could fit either CorpseFinder or AppCleaner.
If the files are inside the Amazon App/Underground App it's likely within the AppCleaners scope.

To add this I need some details on the files to remove (paths, size, when they appear etc.).

We could start by pathdumping the Amazon apps directory structure (Explorer option "Save directory structure"), get an overview 😄.

I wonder if there are more opportunities to clean up waste

There are lots of opportunities, someone just has to take a deep look into apps and their behavior. There are just too many apps for me to do this alone 😉 .

@Gitoffthelawn

This comment has been minimized.

Gitoffthelawn commented Aug 6, 2016

Sounds good. I have added this to my projects list.

I'm overbooked right now, but I now have it on my list and will hopefully have some info for you within a few weeks. If you don't hear back, just add a ping to this thread, and I'll work towards making it a priority item.

@d4rken d4rken added this to the Next Tasks milestone Aug 28, 2016

@d4rken

This comment has been minimized.

Owner

d4rken commented Sep 5, 2016

Any update on this?

@Gitoffthelawn

This comment has been minimized.

Gitoffthelawn commented Sep 5, 2016

It's been a month already? Ugh, I can't believe it. Things are so busy. I've moved this issue up on my list by a few notches, but still overbooked right now. Sorry. Don't hesitate to ping again. It's on my list, but knowing you are still interested keeps me motivated to get to it! 😄

@Gitoffthelawn

This comment has been minimized.

Gitoffthelawn commented Sep 9, 2016

I'm looking at my schedule, and it may be months before I can work on this. Sorry, I'm just overbooked right now. Would you like me to close this, or would you prefer to leave it open?

@d4rken

This comment has been minimized.

Owner

d4rken commented Sep 9, 2016

leave it open

@d4rken d4rken closed this Sep 9, 2016

@d4rken d4rken reopened this Sep 9, 2016

@d4rken d4rken modified the milestones: Blue Sky, Next Tasks Sep 9, 2016

@Gitoffthelawn

This comment has been minimized.

Gitoffthelawn commented Oct 8, 2016

Just wanted to let you know haven't forgotten about this, but I'm still overbooked. Things are just real busy, and also I don't have an extra test device that is available right now. Sorry.

@Gitoffthelawn

This comment has been minimized.

Gitoffthelawn commented Dec 7, 2016

I finally had some time to look into this and determine what's happening.

When the Amazon app fails to install any apk file properly, it places it in a folder named apks in the default storage path. This folder quickly fills up with various apk files that have failed to install (due to storage constraints or other issues).

The cleanup by SDMaid will involve detecting if a folder named apks exists in the default storage path, and removing (with user consent) any apks in that folder, as well as the folder itself.

I tested the Amazon app for this issue, and it had left over 500MB of apk files in that folder.

I hope this helps you and SDMaid! :)

@d4rken

This comment has been minimized.

Owner

d4rken commented Dec 7, 2016

The cleanup by SDMaid will involve detecting if a folder named apks exists in the default storage path, and removing (with user consent) any apks in that folder, as well as the folder itself.

What do you mean with "user consent"?
The initial idea was to have the AppCleaner find the files, if the "Hidden cache" filter is enabled. "Consent" would be the user confirming the deletion for the Amazon entry in the AppCleaner. Did you mean that, or did you mean an additional consent?

Could you use the Explorer option "Save directory structure" on the apks folder (preferably with it having content) and then attach the logfile to this ticket?

@Gitoffthelawn

This comment has been minimized.

Gitoffthelawn commented Dec 7, 2016

What do you mean with "user consent"?
The initial idea was to have the AppCleaner find the files, if the "Hidden cache" filter is enabled. "Consent" would be the user confirming the deletion for the Amazon entry in the AppCleaner. Did you mean that, or did you mean an additional consent?

Yes. :) I do mean what you wrote: "Consent" would be the user confirming the deletion.

Could you use the Explorer option "Save directory structure" on the apks folder (preferably with it having content) and then attach the logfile to this ticket?

I'm not seeing any Explorer option "Save directory structure". Do you know why?

The apks folder is flat: there are no subfolders within it.

@d4rken

This comment has been minimized.

Owner

d4rken commented Dec 7, 2016

I'm not seeing any Explorer option "Save directory structure". Do you know why?

See Explorer settings.

@Gitoffthelawn

This comment has been minimized.

Gitoffthelawn commented Dec 7, 2016

Will do. I'll take a look.

@jawz101

This comment has been minimized.

jawz101 commented Dec 16, 2016

Whenever I try to save directory structure it gives a message that it failed and in my sdcard it has saved a bunch of zero byte files named like folder paths.

Regardless, Amazon Underground stores apks in
/data/data/com.amazon.mShop.android
files/

Prior to installation but there are other files and folders in there as well. TBH, the Amazon Underground app is one of the most complex and analytic oriented apps out there. Like, it's constantly trying to read logs, upload metrics... all sorts of crap.

Download link

https://www.amazon.com/gp/mas/get/android/

It's basically the same as the Google Play Amazon app but they made the Underground app as a sideload which simply adds access to their own app store- since Google kicked the Amazon app off if Google Play Store for a while for competing.

@d4rken

This comment has been minimized.

Owner

d4rken commented Dec 16, 2016

@jawz101 With a full dump of the directory structure we might be able to add some stuff to appcleaners analytics filter too 😁

Regarding "save directory structure" failing, could repeat this while recording a debug log and make a ticket for it?

@jawz101

This comment has been minimized.

jawz101 commented Dec 16, 2016

done. at least the ticket thing. I'll do the directory structure thing once that gets fixed unless I can just do some ls command from the terminal.

@d4rken d4rken modified the milestones: Next Tasks, Blue Sky Dec 17, 2016

@jawz101

This comment has been minimized.

jawz101 commented Dec 18, 2016

I emailed you the Amazon Underground App directory structure.
I disabled any ad blocking, enabled all permissions so it could log whatever it wants from me, and browsed around the app for a while.

I then saved directory structure.
Next, I ran SD Maid app cleaner and saved another copy of the directory structure.

The only thing this wouldn't represent is the part of the ticket where if an app is downloaded but not installed. In this situation apk's would accumulate in the directory I previously mentioned.

Oh, and the Amazon app in the Play Store actually uses the same directory structure but symlinks to this package folder if both the Amazon Play Store app and the Underground app (dl'd from Amazon.com) are installed.

com.amazon.mShop.android.shopping (play store version)

Basically, if you plan on making some custom cleaning stuff it'll mostly work on either folder.

@d4rken

This comment has been minimized.

Owner

d4rken commented Dec 18, 2016

Thanks for the files. In the "after" file, there is nothing I would say also needs to be deleted. No hidden caches or anything like that, basically just databases and config files.

So what's left is the cleaning of downloaded apks. When do those need to be removed or can those be removed everytime they are found? Meaning just delete folder content, or do we need custom logic?
(I did not yet fully understand how it works)

@Gitoffthelawn

This comment has been minimized.

Gitoffthelawn commented Dec 19, 2016

The apks folder and its contents can be deleted, assuming there are only apk files in that folder. However, since apks is not an exceptionally unique name, I suppose false positives could exist. For example, someone could simply have a folder named apks. But I believe the user is always presented with a screen of what will be deleted prior to deletion.

So one fail would be when a user was advanced enough to manually make a folder named 'apks, download apk files and place them in that folder, and then not see that folder listed within SD Maid's UI. Another fail would be when another app creates a folder named apks`. I have never seen that happen, but there are tens of thousands of apps, and I haven't tried all of them. ;)

@d4rken

This comment has been minimized.

Owner

d4rken commented Dec 19, 2016

Is there any way we can recognize that the apks folder is from Amazon, by it's content? Should delete all apks in there or just those that are installed or not installed?

@Gitoffthelawn

This comment has been minimized.

Gitoffthelawn commented Dec 19, 2016

Is there any way we can recognize that the apks folder is from Amazon, by it's content?

I don't think it's possible to recognize it as from Amazon (well, without creating a system to monitor the Amazon app, which would be too much work IMO). Amazon does not create any hidden files or anything like that to indicate it is the owner.

Should delete all apks in there or just those that are installed or not installed?

Delete them all! Amazon has a bug where it does not clean up properly after itself. Delete the folder too. Amazon will recreate it if it needs it again.

@d4rken

This comment has been minimized.

Owner

d4rken commented Dec 20, 2016

/data/data/com.amazon.mShop.android/files/*.apk

Oh i thought it was on public storage... well this makes it much easier, no chance that other apps interfere here.

Can we get a pathdump of the amazon app with a few apks in the folder?

@jawz101

This comment has been minimized.

jawz101 commented Dec 20, 2016

emailing you in a minute with a path dump. Downloaded a bunch of apps and didn't install so they're just hanging out in the folder.

@d4rken

This comment has been minimized.

Owner

d4rken commented Dec 20, 2016

@jawz101

This comment has been minimized.

jawz101 commented Dec 20, 2016

it's not letting me. I did email it though.

Regardless, I looked through and tested deleting all cache directories or where they gather analytics. I deleted all of these directories and they just recreated the next time I started the app and didn't log me out.
the only one that was req'd to keep me logged in was:
/data/data/com.amazon.mShop.android/databases

get recreated:
/data/data/com.amazon.mShop.android/app_ad3.report.venezia
/data/data/com.amazon.mShop.android/app_ad3-meta
/data/data/com.amazon.mShop.android/app_bundle-store
/data/data/com.amazon.mShop.android/app_database
/data/data/com.amazon.mShop.android/app_engagement
/data/data/com.amazon.mShop.android/app_gamelab
/data/data/com.amazon.mShop.android/app_HIGH
/data/data/com.amazon.mShop.android/app_manifest-store
/data/data/com.amazon.mShop.android/app_NORMAL
/data/data/com.amazon.mShop.android/app_RESERVED_FOR_NON_ANONYMOUS_METRICS
/data/data/com.amazon.mShop.android/app_ssnap-cert-store
/data/data/com.amazon.mShop.android/app_textures
/data/data/com.amazon.mShop.android/app_venezia
/data/data/com.amazon.mShop.android/app_venezia-metrics
/data/data/com.amazon.mShop.android/app_weblab
/data/data/com.amazon.mShop.android/app_webview
/data/data/com.amazon.mShop.android/app_zeroes
/data/data/com.amazon.mShop.android/cache
/data/data/com.amazon.mShop.android/code_cache
/data/data/com.amazon.mShop.android/files
/data/data/com.amazon.mShop.android/shared_prefs

@d4rken

This comment has been minimized.

Owner

d4rken commented Dec 20, 2016

https://help.github.com/articles/file-attachments-on-issues-and-pull-requests/

Hm should work though. Don't the pathdumps end in .txt?
I always need more than just the pathes (ownership, file type, permissions etc), SD Maid want's to be as precise as possible. 😁

I don't see any .apk's in your listing either?

Regardless, I looked through and tested deleting all cache directories or where they gather analytics.

Which folders specifically? We could extend the analytics filter SD Maid has in the AppCleaner just for the Amazon app. How did you determine they were analytics data?

@jawz101

This comment has been minimized.

jawz101 commented Dec 20, 2016

try this. My Firefox NoScript add-on was blocking the drag & drop script on here.

pathdump#_data_data_com.amazon.mShop.android#1482251820168.txt

there should be a bunch of files in the attachment called update....apk
/data/data/com.amazon.mShop.android/files/update-3-1482249801457.apk

@d4rken

This comment has been minimized.

Owner

d4rken commented Dec 20, 2016

So the pattern is always update-X-XXXXXXX.apk, should we match against that, or against *.apk? Could there be *.apk files that shouldn't be deleted? According to you guys not... On the otherside if we match against *.apk and amazon changes the naming pattern, then the matching doesn't break...

So whats the analytics ones?

On first view I see:

DIRECTORY  771 10189:10189    4.00 KB   20/12/16 10:03:23 /data/data/com.amazon.mShop.android/app_venezia-metrics
FILE       600 10189:10189   28.00 KB   20/12/16 10:08:33 /data/data/com.amazon.mShop.android/app_venezia-metrics/availability_measurements_1482200315056.measure

and

DIRECTORY  771 10189:10189    4.00 KB   20/12/16 10:18:40 /data/data/com.amazon.mShop.android/app_ad3-meta
FILE       600 10189:10189    4.00 KB   20/12/16 10:18:40 /data/data/com.amazon.mShop.android/app_ad3-meta/availability_measurements_1482200316719.measure

anything else=

@jawz101

This comment has been minimized.

jawz101 commented Dec 20, 2016

I think any .apk file. If an app is successfully installed the apk file is removed, so this would fix those instances when you download an app but choose not to install it or if a download is interrupted.

You are correct on the paths for the analytics. Those are the ones it's collected on me, though, Amazon has several methods where they try to gather stuff on the user.

I'll have to enable all permissions they request and turn off ad blocking for a while to see if other stuff gets logged.

Maybe hold off for a day while I test some more- but those 2 spots you mentioned are at least 2 analytic folders/files that do collect stuff.

@Gitoffthelawn

This comment has been minimized.

Gitoffthelawn commented Dec 20, 2016

Amazon will create apk's with any filename in the apks folder within public storage. So the match would be *.apk and then delete the folder if it is empty.

Yep, another app could save an apk there, although I've never encountered that.

@d4rken

This comment has been minimized.

Owner

d4rken commented Dec 20, 2016

I'm confused, you are talking about public storage now?

The files pathes posted above are on private storage, and not in an apks folder but bare in the files folder.

@Gitoffthelawn

This comment has been minimized.

Gitoffthelawn commented Dec 20, 2016

Let's make sure we are defining terms the same way... what's your definition of public storage?

@d4rken

This comment has been minimized.

Owner

d4rken commented Dec 20, 2016

Public storage is storage that is accessible by all apps, and accessible without root, this includes:

  • a public internal sdcard (e.g. /storage/emulated/0)
  • a public external sdcard (e.g. /storage/sdcard1)
  • and possibly public portable storage (e.g. /storage/usb)

Private storage holds apps private data that is only accessible to them and is only accessible with root.

  • In almost all cases the primary private storage under (e.g. /data)
  • On some newer devices 6.0 devices with adopted storage also a secondary private storage (e.g. /mnt/expand/....)
@Gitoffthelawn

This comment has been minimized.

Gitoffthelawn commented Dec 20, 2016

OK, perfect. Our definitions are identical! :)

I am referring to items in public storage.

@jawz101 is, I believe, referring to items in private storage.

@jawz101

This comment has been minimized.

jawz101 commented Dec 20, 2016

I don't think it's saving anything saves to public storage, from what I can tell. Well, Amazon Prime Video, Music and Kindle eReader may cache offline media there but Amazon cleans these up themselves since they're DRM files I think.

This time I installed a few of the other Amazon apps (Kindle, Music, Drive, Alexa, Prime Photos, Prime Video), granted them access to storage, location, phone,... even gave Amazon Underground access to "observe other apps"- I don't see many apps with that permission aside from antivirus programs.
screenshot_20161220-163948

Ran them for a while and let them gather junk.

Next, downloaded a few Amazon games and played them for a while to gather info as well.

Finally, cleaned with SD Maid App and System Cleaner.

Here's the results:
pathdump#_data_data_com.amazon.mShop.android#1482274274437.txt

I think that's my last effort on this but as you can see it added a few more *.measure files sprinkled throughout the directories.

@d4rken

This comment has been minimized.

Owner

d4rken commented Dec 20, 2016

@jawz101 Good pathdump! So we can add the *.measure files under the analytics filter and the update-*.apk files under the HiddenCache filter.

@Gitoffthelawn Still need a pathdump of the apks folder on public storage you keep mentioning. Are you sure it's from Amazon? Why would it store apks in private AND public storage?

@jawz101

This comment has been minimized.

jawz101 commented Dec 20, 2016

that's what I was thinking as well. Those 2 filters would be a nice, concise choice.

I have both internal public storage an an SD card in my device- haven't seen apk's download from the Amazon App to either "public" storage location. There are just empty /sdcard/Android/data/com.amazon.... folders out there like most apps make.

@jawz101

This comment has been minimized.

jawz101 commented Dec 21, 2016

FYI

ahh... now I see. If I grant Amazon Underground Storage permission it saves the apk's out to private storage. It never prompted to but I had to intentionally go in and grant Storage permissions in Android's settings screen.

/storage/emulated/0/Android/data/com.amazon.mShop.android/files/apks/

if Amazon Underground does not have Storage permission it saves to public storage
/data/data/com.amazon.mShop.android/files/

so I guess either spot is a potential location for cleaning.

@d4rken

This comment has been minimized.

Owner

d4rken commented Dec 21, 2016

Why does it use a subfolder one but not the other? Weird.

What else is saved on public storage if permission is granted? Just APKs?

@jawz101

This comment has been minimized.

jawz101 commented Dec 22, 2016

as far as I can tell, yes, but I didn't take it too far after that. I just wanted to see if that storage permission was why his apk's might be storing in another spot but I didn't play around in the app to see if other junk would end up there :/

@d4rken

This comment has been minimized.

Owner

d4rken commented Dec 23, 2016

Alright, let's do this.

We had a nice discussion here so I wanted to also share what we ended up with.

For the apks, the filter looks like:

{
  "packages": ["com.amazon.mShop.android"],
  "fileFilter": [
	{
	  "locations": ["PRIVATE_DATA"],
	  "contains": ["com.amazon.mShop.android/files/update-"],
	  "patterns": ["^(com\\.amazon\\.mShop\\.android\\/files\\/update-\\d+-\\d+\\.apk)$"]
	}, {
	  "locations": ["PUBLIC_DATA"],
	  "contains": ["com.amazon.mShop.android/files/apks/update-"],
	  "patterns": ["^(com\\.amazon\\.mShop\\.android\\/files\\/apks\\/update-\\d+-\\d+\\.apk)$"]
	}
}

which satisfies this test:

    @Test
    public void testAmazonMarket_downloadedApks() throws Exception {
        addPositive("com.amazon.mShop.android/files/update-1-1.apk", Location.PRIVATE_DATA);
        addPositive("com.amazon.mShop.android/files/update-12-123456789.apk", Location.PRIVATE_DATA);
        addNegative("com.amazon.mShop.android/files", Location.PRIVATE_DATA);
        addNegative("com.amazon.mShop.android/files/some.apk", Location.PRIVATE_DATA);
        addNegative("com.amazon.mShop.android/files/update-1-12346789.zip", Location.PRIVATE_DATA);
        addNegative("some.other.app/files/update-1-1.apk", Location.PRIVATE_DATA);

        addPositive("com.amazon.mShop.android/files/apks/update-1-1.apk", Location.PUBLIC_DATA);
        addPositive("com.amazon.mShop.android/files/apks/update-12-123456789.apk", Location.PUBLIC_DATA);
        addNegative("com.amazon.mShop.android/files/apks", Location.PUBLIC_DATA);
        addNegative("com.amazon.mShop.android/files/apks/some.apk", Location.PUBLIC_DATA);
        addNegative("com.amazon.mShop.android/files/apks/update-1-12346789.zip", Location.PUBLIC_DATA);
        addNegative("some.other.app/files/apks/update-1-1.apk", Location.PUBLIC_DATA);

        confirm(new HiddenFilter(getSDMContext()), "com.amazon.mShop.android");
    }

and the analytics filter looks like this:

{
"packages": ["com.amazon.mShop.android"],
  "fileFilter": [
	{
	  "locations": ["PRIVATE_DATA"],
	  "contains": [
		"com.amazon.mShop.android/app_engagement",
		"com.amazon.mShop.android/app_gamelab",
		"com.amazon.mShop.android/app_venezia-metrics",
		"com.amazon.mShop.android/app_ad3-meta"
	  ],
	  "patterns": ["^(com\\.amazon\\.mShop\\.android\/app_[\\w_\\-]+\/\\w+_measurements_\\d+\\.measure)$"]
	}
  ]
}

and satisfies this test:

    @Test
    public void testAmazonMarket_analytics() throws IOException, InterruptedException {
        addDefaultNegatives();

        addNegative("com.amazon.mShop.android/app_venezia-metrics", Location.PRIVATE_DATA);
        addPositive("com.amazon.mShop.android/app_venezia-metrics/availability_measurements_1482267297904.measure", Location.PRIVATE_DATA);
        addPositive("com.amazon.mShop.android/app_venezia-metrics/thecakeisalie_measurements_4444444444.measure", Location.PRIVATE_DATA);

        addNegative("com.amazon.mShop.android/app_gamelab", Location.PRIVATE_DATA);
        addPositive("com.amazon.mShop.android/app_gamelab/availability_measurements_1482267313334.measure", Location.PRIVATE_DATA);
        addPositive("com.amazon.mShop.android/app_gamelab/something_measurements_1111.measure", Location.PRIVATE_DATA);

        addNegative("com.amazon.mShop.android/app_engagement", Location.PRIVATE_DATA);
        addPositive("com.amazon.mShop.android/app_engagement/availability_measurements_1482267758696.measure", Location.PRIVATE_DATA);
        addPositive("com.amazon.mShop.android/app_engagement/something_measurements_2222222.measure", Location.PRIVATE_DATA);

        addNegative("com.amazon.mShop.android/app_ad3-meta", Location.PRIVATE_DATA);
        addPositive("com.amazon.mShop.android/app_ad3-meta/availability_measurements_1482256138880.measure", Location.PRIVATE_DATA);
        addPositive("com.amazon.mShop.android/app_ad3-meta/strawberry_measurements_3333333333.measure", Location.PRIVATE_DATA);

        confirm(new AnalyticsFilter(getSDMContext()), "com.amazon.mShop.android");
    }

Committed for the next update.

@d4rken d4rken closed this Dec 23, 2016

@d4rken d4rken modified the milestones: v4.5.5, Next Tasks Dec 23, 2016

@d4rken

This comment has been minimized.

Owner

d4rken commented Dec 23, 2016

If you would like to try it, send a mail to support@...

@Gitoffthelawn

This comment has been minimized.

Gitoffthelawn commented Jan 16, 2017

As of v4.5.9, it looks like this is implemented in AppCleaner, and not SystemCleaner. The matched files are so large, I wonder if they should be included in SystemCleaner. What do you think?

@d4rken

This comment has been minimized.

Owner

d4rken commented Jan 16, 2017

Why should we add this to the SystemCleaner instead of AppCleaner?
What does the size have to do with it? The SystemCleaner is intended for items that can't be matched to specific apps, i.e. ROM logs, public thumbnails.

@Gitoffthelawn

This comment has been minimized.

Gitoffthelawn commented Jan 17, 2017

Good questions. I guess I didn't understand that distinction, but that does make sense.

It gets a bit gray because SystemCleaner cleans temp files and thumbnails specific to individual apps.

Also, the size of the APK's left behind by Amazon can quickly cause the entire System to be unusable.

But since the APK's are created by an individual app (Amazon) and are often (but not always) left in Private Storage, AppCleaner does also make sense.

@d4rken

This comment has been minimized.

Owner

d4rken commented Jan 17, 2017

It gets a bit gray because SystemCleaner cleans temp files and thumbnails specific to individual apps.

Yeah there is a bit of overlap, because tools are preferably kept independent, it's unavoidable to prevent both tools matching the same files, but really is no issue either.
This could also happen with CorpseFinder and Duplicates etc.

SD Maid cares for these overlaps by checking for failed deletions, SD Maid checks if the deletion failed because the file no longer exists.

Idealy we would want the AppCleaner to cover all of the files the SystemCleaner targets because the AppCleaner can deal with package based exclusions, check if apps are running, kill them before deleting files if necessary and so on. Just got more options here.

But for stuff like ROM files or unknown apps, and also to allow user configured filters, the SystemCleaner is necessary. There is no "app" or something else that's reliably detectable to which we could match kernels/ROM/recovery related files.

@Gitoffthelawn

This comment has been minimized.

Gitoffthelawn commented Jan 17, 2017

That sounds good.

One option would be to add a "System" app to AppCleaner that catches all the non-app-specific files to be deleted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment