The repository is too big #119
Comments
|
Perhaps using something like this? --> https://rtyley.github.io/bfg-repo-cleaner/ While searching the This output is provided by adding |
|
We can also take this opportunity and delete some stale branches: I am talking about the lower 4 in the dropdown menu of the image. Any good reason not to clean that up? What exactly are those |
|
I think the purging of the history should be done after the iEEG and EEG branches are merged. This will simplify this whole procedure.
|
Quick summary:In the past, we accidentally pushed huge files to the repository. They were purged in #124 ... but they are still in the git history. ProblemThe repository is ~10 times as big as it should be (570MB) ... so it's slow for cloning I see the following solutions:
This is a big question, so we definitely need input from several people on this. @effigies @yarikoptic @tyarkoni @robertoostenveld @choldgraf @dorahermes |
|
I'm a fan of options 2 and 3 - I think having a repository this large makes it prohibitive for most people to download unless they're really motivated to do so... |
|
Agree with @choldgraf. (2) is pretty extreme though. I guess we could start with (3) and fall back on (2) if things go awry or it turns out to be more difficult than anticipated. |
|
(3) sounds good. There are only two PRs to rebase onto the new history, so helping people (one of them being me...) do that will not be a huge burden. |
|
I wondered (since never used it myself) if "git graft" mechanism could be used in addition to 3 to mark some commit(s) in the past of the old "heavy" history to match corresponding ones in the "new light" history. That should (theoretically, if my understanding is correct) to allow people to proceed with their existing clones/histories as "nothing has happened" while new objects from the new history would come to replace old(er) ones, and new clones would be lightweight. Also not sure how tags would behave etc. BUT given the nature/purpose of bids-examples, I think that pure "3" would be just fine ;) |
|
I also prefer 3. If "git graft" does not work, it would be nice (but not required) to give instructions for others how to resolve/clean their clones. The easy solution that I would probably take is to delete my fork and all local clones and make a new one. Instructing the 54 people that now have a fork on github (see members) would already help a lot. |
|
Alright, I'll then attempt number 3 (BFG repo cleaner) in the next days. I already put a branch protection for In addition, I agree with @robertoostenveld that it might be a good idea to notify all people who have forked the repository. Let's see how this goes! |
Progress so far:
so far so good, the
However ...the push gave some errors, click on details: ... apparently, this is a known problem, see e.g., this SO post, and this issue on BFG: rtyley/bfg-repo-cleaner#36 The problem is that we cannot clean the history that was introduced to our repo via other remotes (PRs) I see three solutions:
I am in favor of 1. or 2. Any opinions, hints, comments? :-) |
|
I'm in favor of:
Thoughts? |
|
+1
… On 12 May 2019, at 20:18, Chris Holdgraf ***@***.***> wrote:
I'm in favor of:
Rename this repository to something like bids-examples-old and keep the history for archival purposes.
Start fresh with a new repository that lives at bids-examples. Create a "what happened to this repository's history?" section that links to the archived version.
Thoughts?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#119 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAG3PYYGZEC7FVG7ZVUFH4TPVBNQRANCNFSM4FRYSQKA>.
|
|
@choldgraf That sounds reasonable to me. There was also discussion in #159 about removing some of the example datasets. Does it make sense to do that during the move, too? |
|
@effigies my 2 cents: while I'm a fan of removing some example datasets, I'd decouple that action from the action of cleaning up history and reducing the size of the repo...it's hard enough making decisions in a distributed fashion :-) |
UpdateTL;DR: BFG worked, repo is now small, history rewritten, everybody delete their forks/clones and make fresh ones Although I initially liked the suggestion of making a new repo, I got worried because I remembered that this might have some consequences for the As a result, I desperately looked at my process again to find issues ... and there you go 🤦♂️ In #119 (comment), it did not work, because the branch protection was set to "on" and the settings didn't allow anyone to push. On the good side: I disabled branch protection and then the cleaning worked very well. Our git history is ~80MB, and the repository with all files ~130MB --> feel free to check by making a fresh clone: Another good thing: Our tags and releases are not affected. The only issue that remains is that the GitHub references of all PRs that ever happened to this repository are still intact (albeit now pointing to a re-written history). The only way to solve it is by contacting GitHub support, asking them to delete those refs ... but I don't see a reason to do that, because we don't have sensitive data in these refs. As long as we don't reopen and merge these old PRs, everything will be fine. On that note: everybody needs to delete their old forks and clones ... and we need to be extra diligent with merging PRs, always making sure that they are based on our new history. |
|
PS: Next up, I'll make a new release of the bids-examples, which will include the EEG and iEEG data Feel free to comment if you have concerns, see: #158 |
We should zero the large MEG files and purge the history.
The text was updated successfully, but these errors were encountered: