History Rewriting to remove SGX Private Key

[tqchen]

Instruction to sync to master after rewrite

If you do not have local changes, you can simply do git reset --hard [HASHTAG-OF-NEW-HEAD]

Please use the following commands

# Add upstream if you have not yet done so
git remote add upstream https://github.com/dmlc/tvm
git remote add prefilter https://github.com/tqchen/tvm
git fetch upstream
git fetch prefilter
# checkout a local branch for safety reason, 
# you can always reset your master to the old head
git checkout -b backup
git checkout master
# Bring your changes on top of the most recent prefilter branch
git rebase prefilter/prefilter
# sync your local changes to master
git rebase --onto upstream/master e316f03d2d2e0c06019b6d026b9696b7d3f67b8d
# force push to your upstream master
git push --force

Here e316f03d2d2e0c06019b6d026b9696b7d3f67b8d is the hashtag of the head before fiilter. The last command will take the commits between ``e316f03``` and your head and apply it to the upstream/master's HEAD

Context

The history has been rewritten

• An old history branch is kept at branch prefilter, and we will delete it after 0.4

This is to followup issue on #1189

So far in the SGX demo, we introduced private keys into the commit history. Although the private key is only used for demo purposes and poses no security concern, it triggers false alarm of security scanning tools.

Currently, the private keys are shallowed removed. But it can still trigger the alarm due to the fact that the file is still in git history.

I have created https://gist.github.com/tqchen/ca5f1b898e27035621130d87aa9bebaf to deeply filter the history to remove the file. With an example branch here https://github.com/tqchen/tvm/commits/filter that gives the result of filter-branch.

Note that filter-brach causes divergence from the current tree, and in order to bring this change to master, we have to force rewrite the history and push to master -- note that commits contributions are preserved, but it indeed will cause troubles for our contributors.

This is a decision that can not be made lightly. So this is an RFC post, to hear opinions from the community on whether we should do this or not. Please express your opinions in this thread.

This thread will remain open for one week before we reach a decision

Pros and Cons

• Pros: avoid security check alarm in the future, although it is a false alarm.
• Cons: require contributors to sync up with the master via history rewriting.

[tqchen]

@dmlc/tvm-team @mnuyens

[alex-weaver]

Regardless of whether this is actually carried out or not, I think the most important thing is (if possible) add a check to the CI build to prevent security keys from ever being merged in the future. This would at least prevent needing to make this same decision again.

[yzhliu]

Agree with @alex-weaver that we should add checker to CI.
Personally I support to remove the key file from history since this can confuse people who are sensitive to security and we may lose our potential users.

[mnuyens]

This doesn't remove all cases, it also needs to be removed from the history of the python file.

I'm working on prepping a branch that has it removed as well.

[tqchen]

@mnuyens can you comment which specific python file that we need to remove?
It would be great if you can help to update https://gist.github.com/tqchen/ca5f1b898e27035621130d87aa9bebaf to be the one that contains the correct form

Because the master branch is still evolving, we will need an automatic script to do the filtering, Thanks!

[mnuyens]

We don't need to remove a whole file just some lines from apps/sgx/prepare_test_libs.py.
I'll test out changes to the command today and upload one that I think works along with a example branch that I've run the command on to confirm that it still keeps everything the way it should be.

[mnuyens]

Here is my command that both removes the pem file and removes the necessary lines from apps/sgx/prepare_test_libs.py: https://gist.github.com/mnuyens/78deb8c9ad3df0070e0514ce62717659. I'll update with the new branch once I get that up.

[mnuyens]

Here is the branch I've made a sample: https://github.com/mnuyens/tvm/tree/revised. I'm going confirm with the security team that this solves any problems and will update with confirmation once that's done.

[tqchen]

@mnuyens can you confirm if your script works?

[mnuyens]

Still working with to security to confirm sorry for the delay.

[mnuyens]

Go ahead with it.

[tqchen]

After deliberation and discussion with the team, we are going to filter the history to remove the trace. This is a reminder that we are going to do the filtering branch in 24 hours and please rebase your code to master after it gets updated @dmlc/tvm-team

[tqchen]

The master branch has been filtered and put back into protection mode, please rebase your code against master

[tqchen]

Please see the beginning of the post on how to sync back to the current master

[tqchen]

close this PR since rewriting is done

[tqchen]

As a note, the prefilter branch is now deleted from the current main repo, it will be kept in my fork https://github.com/tqchen/tvm for a while, instruction has been updated