Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decrease the download size of the repo. #2820

Closed
yashk2000 opened this issue Jul 9, 2019 · 59 comments

Comments

@yashk2000
Copy link
Member

@yashk2000 yashk2000 commented Jul 9, 2019

Describe the chore
The current download size of the repository is very large, 292.08mb. It should be decreased as the code size in the repo is barely around 20mb.
This will be done using bgf.

Would you like to work on the issue?

  • Yes
  • No
  • Other:
@yashk2000 yashk2000 added this to the Milestone 7: Summer 2019 milestone Jul 9, 2019
@yashk2000 yashk2000 changed the title Decease the download size of the repo. Decrease the download size of the repo. Jul 10, 2019
@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 10, 2019

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 10, 2019

git rev-list --objects --all | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | sed -n 's/^blob //p' | sort --numeric-sort --key=2 | cut -c 1-12,41- | numfmt --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 10, 2019

I ran the command.
256e4355e77a 6.2MiB sample-apk/app-debug-unaligned.apk
a6fa80a6e144 6.2MiB sample-apk/app-debug-unaligned.apk
2e6d33385169 6.2MiB sample-apk/app-debug-unaligned.apk
c28a3bf0975e 6.2MiB sample-apk/app-debug-unaligned.apk
7a71ec4c6cca 6.2MiB sample-apk/app-debug-unaligned.apk
870c8e469983 6.4MiB sample-apk/app-debug-unaligned.apk
c73398da9692 7.1MiB app/build/outputs/apk/app-debug-unaligned.apk
4caacd1d0019 7.1MiB app/build/outputs/apk/app-debug-unaligned.apk
727d9c562683 7.1MiB app/build/outputs/apk/app-debug.apk
60bcfdde62d2 7.1MiB app/build/outputs/apk/app-debug.apk
3033a4996755 13MiB app/build/intermediates/instant-run-support/debug/classes.zip
975df380d853 13MiB app/build/intermediates/instant-run-support/debug/classes.zip
63e125ed8d95 22MiB build/generated/mockable-Google-Inc.-Google-APIs-23.jar
c6bf479cc786 22MiB build/generated/mockable-android-23.jar
5edbc38e9d27 24MiB .gradle/2.10/taskArtifacts/fileSnapshots.bin
0fef89a5dd96 24MiB .gradle/2.10/taskArtifacts/fileSnapshots.bin
44a8bdb32c4b 30MiB phimpme-dev-app-release-unsigned.apk
73c675882045 30MiB phimpme-master-app-release-unsigned.apk
4cf1f088501b 31MiB phimpme-master-app-release-unaligned.apk
ae4d36ed6094 31MiB phimpme-master-app-release.apk
a7e1342d5e00 34MiB phimpme-dev-app-debug.apk
8a00c89d4de4 34MiB phimpme-master-app-debug.apk

Should I remove these?

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 10, 2019

Your analysis includes the apk branch, delete it and rerun

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 11, 2019

@iamareebjamal I did that, I'll remove some media files also which seem to be increasing the repo size.
Currently, this is being shown after I pushed:
This branch is 1658 commits ahead, 1658 commits behind fossasia:development.
Is it alright?
https://github.com/yashk2000/phimpme-android/tree/repo_size
This is the branch.

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 11, 2019

Yes it is right. But you'll first give the list of files to remove, I'll review it and move forward. Every contributor will have to reclone so it's not straightforward

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 11, 2019

sample-apk
app-debug-unaligned.apk
phimpme-master-app-debug.apk
phimpme-dev-app-debug.apk
phimpme-master-app-release.apk
phimpme-master-app-release-unaligned.apk
phimpme-master-app-release-unsigned.apk
phimpme-dev-app-release-unsigned.apk
fileSnapshots.bin
mockable-Google-Inc.-Google-APIs-23.jar
ss_7.png
73-usersthink-stock-image-web.jpg
ss_8.png
screenshot_5.png
oc_sample.png
31-usersthink-stock-image-web.jpg
58-usersthink-stock-image-web.jpg
commons-lang3-3.1-javadoc.jar
ss_1.png
ss_2.png
ss_4.png
ss_5.png
classes.dex
ss_6.png
merger.xml
75-usersthink-stock-image-web.jpg
PhimpMe.apk
resources-debug.ap_
91-usersthink-stock-image-web.jpg
37-usersthink-stock-image-web.jpg
resources.zip
71-usersthink-stock-image-web.jpg
app-debug.apk
classes.zip
mockable-android-23.jar

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 11, 2019

@iamareebjamal these are the files.
There are several more files that can be removed, but it is too tedious to go through the list and find out each and every small file that can be removed.

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 11, 2019

Cloning into 'phimpme-android'...
remote: Enumerating objects: 4363, done.
remote: Counting objects: 100% (4363/4363), done.
remote: Compressing objects: 100% (2178/2178), done.
remote: Total 30581 (delta 2610), reused 3916 (delta 2176), pack-reused 26218
Receiving objects: 100% (30581/30581), 59.02 MiB | 585.00 KiB/s, done.
Resolving deltas: 100% (16994/16994), done.

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 11, 2019

phimpme-master-app-debug.apk
phimpme-dev-app-debug.apk
phimpme-master-app-release.apk
phimpme-master-app-release-unaligned.apk
phimpme-master-app-release-unsigned.apk
phimpme-dev-app-release-unsigned.apk

You have apk branch in your repo
git branch -D apk

And rerun

There were a lot of .so, .aar and .html files.

Before this, remove every jar file from here https://github.com/fossasia/phimpme-android/tree/development/app/libs

Very stupid to keep jars in a gradle project when you can just add a dependency in build.gradle

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 11, 2019

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 11, 2019

@iamareebjamal I do not have the apk branch in my repo.

@iamareebjamal

This comment has been minimized.

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 11, 2019

@yashk2000 Then those apks should not be listed

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 11, 2019

@yashk2000 Then those apks should not be listed

I checked using git branch and also did the branch deletion command. The apk branch is not there.

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 11, 2019

OK. We'll see that later. In history, there's entire build folder, so that can be directly removed

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 11, 2019

The pdk module is for pinterest authentication.
I do not know about the photoview module. I think it can be added as a dependency instead.

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 11, 2019

phimpme-master-app-debug.apk
phimpme-dev-app-debug.apk
phimpme-master-app-release.apk
phimpme-master-app-release-unaligned.apk
phimpme-master-app-release-unsigned.apk
phimpme-dev-app-release-unsigned.apk

You have apk branch in your repo
git branch -D apk

And rerun

There were a lot of .so, .aar and .html files.

Before this, remove every jar file from here https://github.com/fossasia/phimpme-android/tree/development/app/libs

Very stupid to keep jars in a gradle project when you can just add a dependency in build.gradle

@iamareebjamal should I delete these jars permanently or using bfg --delete-files ?

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 11, 2019

That won't work. You have to remove the jar from the project and add gradle dependencies in their place and create a PR.

How can you just delete jars without a replacement? And also bfg doesn't allow you to delete anything that is currently in the repo, only from history

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 11, 2019

Should I create a separate pr after this issue is resolved or right now?

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 11, 2019

Read above comment. How will it make a difference after this issue is resolved?

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 11, 2019

Yeah. Ok, doing it.

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 11, 2019

pdk can be made into a gradle dependency as well

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 11, 2019

Ok

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 11, 2019

@iamareebjamal I removed the photoview module and replaced it as dependency. But the pdk module required either the sdk or an aar to be downloaded. So what should I do about the pdk module?

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 11, 2019

I'll give you a link. First send a PR for above stuff

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 12, 2019

Ok.

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 12, 2019

AFAIK, there is one last thing to do before we can move ahead.

https://github.com/fossasia/phimpme-android/commits/gh-pages

gh-pages branch has a history, it shouldn't. It should be like https://github.com/fossasia/phimpme-android/commits/apk

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 13, 2019

@iamareebjamal how to stop the commit history from being recorded for the gh pages branch?

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 13, 2019

@iamareebjamal do we need the github pages? Currently the website, https://phimp.me/ doesn't look very good.

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 13, 2019

http://fossasia.github.io/phimpme-android/

OK, so really weird but the above site is using master branch README.md, not gh-pages. But we do need gh-pages and we need to fix them

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 13, 2019

how to stop the commit history from being recorded for the gh pages branch?

The same way it is done for apk branch

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 15, 2019

@iamareebjamal the current gh pages is continuously updated by yaydoc bot. The branch is not deleted and rebuilt completely. Hence the commit history is recorded.
The apk branch is deleted and rebuilt everytime. Hence the commit history is not there.

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 15, 2019

OK. Yaydoc website is down as well. Open an issue on yaydoc repo to allow repo users to create orphan branches. We'll delete gh-pages branch for now to move forward

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 15, 2019

Also, who are regular contributors besides you? They'll need to be informed about this change as they will be required to reclone their repos

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 15, 2019

OK. Yaydoc website is down as well. Open an issue on yaydoc repo to allow repo users to create orphan branches. We'll delete gh-pages branch for now to move forward

Ok.

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 15, 2019

Also, who are regular contributors besides you? They'll need to be informed about this change as they will be required to reclone their repos

@iamareebjamal currently, I am the only active contributor. @sonusourav created a few issues and mentioned that he'll be working on them, but there has not been any activity yet.

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 15, 2019

Should I proceed with using bgf to clean the commit history now?

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 15, 2019

You can proceed and list the file and folders removed and then push to your fork. I'll review and add some more, remove some and iterate will it's ensured no extra file is being removed and no extra file is remaining, then I'll push to the repo

Remove apk and gh-pages branches before continuing

@yashk2000

This comment has been minimized.

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 15, 2019

fileSnapshots.bin
mockable-android-23.jar
mockable-Google-Inc.-Google-APIs-23.jar
classes.zip
app-debug.apk
app-debug-unaligned.apk
resources.zip
Phimpme.apk
PhimpMe.apk
resources-debug.ap_
ss_7.png
ss_8.png
screenshot_5.png
oc_sample.png
commons-lang3-3.1-javadoc.jar
ss_1.png
ss_2.png
ss_5.png
ss_4.png
ss_3.png
ss_1.png
classes.dex
merger.xml
screenshot_2.png

@fossasia fossasia deleted a comment from yashk2000 Jul 15, 2019
@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 15, 2019

These are way few files and folders. Normally, you won't remove things like this. You just say bfg --delete-folders build, bfg --delete-folders libs, bfg --delete-folders pdk etc. And then delete files on some left out files

What'd be the point of deleting all those jars, modules and files if you're not going to remove it? And it'll take ages if you remove all those files one by one

Make a PR to remove javadocs of openCV, we don't need to maintain that
https://github.com/yashk2000/phimpme-android/tree/development/openCV/javadoc

Then delete that as well

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 16, 2019

.github
Dangerfile
upload-apk.sh
.project
gradle.properties
build.gradle
upload_apk_forks.sh
.travis.yml
gradlew
CODE_OF_CONDUCT.md
CONTRIBUTING.md
lint.xml
viewflow
imageeditlibrary
scripts
.DS_Store
import-summary.txt
LICENSE
AndroidManifest.xml
README.md
res
pdk
openCV
src
openCVLibrary24133
app
.idea
metadata
fastlane
libs
docs
PhimpMe.apk
sample-app
sample-apk
.gradle
build

@iamareebjamal I have deleted these files. Please tell if there are anymore files to be deleted.

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 16, 2019

Dangerfile
upload-apk.sh
.project
gradle.properties
build.gradle
upload_apk_forks.sh
.travis.yml
gradlew
CODE_OF_CONDUCT.md
CONTRIBUTING.md
lint.xml

None of these files are to be deleted. Most of them are in use and don't contribute to the size of the repo

Add a distinction between file and folder

Why delete docs, fastlane, metadata, etc? All of these are in use. I don't think you have understood the process correctly. Delete folders mainly, then if remaining, delete large files. And obviously don't delete files and folders currently present in the repo

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 17, 2019

I thought that the commits of these fiiles at HEAD are preserved, that's why I did that. I'll revert the changes and make a new branch.

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 17, 2019

Folders:
javadoc
screenshots
_static
photoview
openCVLibrary24133
sample-app
sample-apk
.gradle
pdk
libs
build

Files:
oc_sample.png
PhimpMe.apk

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 17, 2019

Will see

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 18, 2019

@iamareebjamal is it alright?

@fossasia fossasia deleted a comment from yashk2000 Jul 18, 2019
@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 18, 2019

jniInclude and others are missing. Didn't check thoroughly though

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 18, 2019

@iamareebjamal do I need to send a pr? Or will you be force pushing directly to the repo?

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 18, 2019

I'll be pushing after complete review

@yashk2000

This comment has been minimized.

Copy link
Member Author

@yashk2000 yashk2000 commented Jul 18, 2019

Ok

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 18, 2019

Final Cleanup List:

o .gradle/
i *.apk
o build
o jniLibs
o _site
o backgrounds
o 3rdparty
o libs
o javadoc
o _static
o screenshots
o .idea
o bower_components
i *.aar
o openCVLibrary24133
i *.pdf
o obj
o include
o photoview
o pdk
i .log
i device-
.png
i oc_sample.png
i awesome-app-landing-page-screenshot.png
i phone.png
i screenshot.png
o img

o - folder
i - file

Repo Size - 32 MB

@iamareebjamal

This comment has been minimized.

Copy link
Member

@iamareebjamal iamareebjamal commented Jul 18, 2019

Done!

git clone https://github.com/fossasia/phimpme-android
Cloning into 'phimpme-android'...
remote: Enumerating objects: 6304, done.
remote: Counting objects: 100% (6304/6304), done.
remote: Compressing objects: 100% (4674/4674), done.
remote: Total 23695 (delta 4285), reused 3539 (delta 1622), pack-reused 17391
Receiving objects: 100% (23695/23695), 27.10 MiB | 6.19 MiB/s, done.
Resolving deltas: 100% (14248/14248), done.

Refork and reclone! 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
2 participants
You can’t perform that action at this time.