Change format of facade directories

[IsaacMilarky]

Description

• Change the repo_path that facade collection used to no longer be determined by the repo's group id, now a repo_id is used which is unique to that repo.
• Create an alembic script to automatically migrate instances to the new change
• Fix issues associated with the aforementioned change due to changes in formatting

Signed commits

• Yes, I signed my commits.

[sgoggins]

This LGTM .. with the assumption that something in here is deleting the old clone, and making a new one ... or moving the old one. Either way, we can't afford to have the old clones and the new clones both on disk. It would be too costly for augur users, and not something they expect.

[sgoggins]

I want @ABrain7710 to review this before we merge it.

[IsaacMilarky]

This LGTM .. with the assumption that something in here is deleting the old clone, and making a new one ... or moving the old one. Either way, we can't afford to have the old clones and the new clones both on disk. It would be too costly for augur users, and not something they expect.

The old clones are deleted and then cloned again via an alembic script.

[ABrain7710]

The only thing I see that needs added is a query that updates the core and secondary statues to pending if the issue_pr_sum is NULL

[IsaacMilarky]

The only thing I see that needs added is a query that updates the core and secondary statues to pending if the issue_pr_sum is NULL

That makes sense. I have made the change

[ABrain7710]

I think we should try to restrict these operations rather than just running them on every repo, so that if someone runs a downgrade/updgrade in the future it won't just reset everything for no reason

[ABrain7710]

I think we should only run this on repos that don't have a commit_count or have a repos that are stored in the old structure. The reason being that I would like this script to reset all of facade on a huge instane if someone accidentally downgrades the database

[IsaacMilarky]

If they wanted to downgrade the schema it would be proper to also reset the facade repos again. We also can't check based on the commit_count because the revision before this one has commit_count without the directory change. I think it's fine how it is in my opinion.

[IsaacMilarky]

I don't think it matters if someone accidentally clears their facade repos while downgrading the schema since it's so fast to clone.

[ABrain7710]

Reply to first part: How would it be proper? I don't see why that would be the desired behavior. Usually in my experience downgrades are to bring the database back to the prior state, but resetting the repos doesn't do anything to reset to the old state.

Reply to second part: I think we can do it based on commit_count we would run it if the facade directory was wrong or the commit_count doesn't exist since they are both valid times to do it.

Reply to third part: I do think it matters, because although it is fast to clone, each repo still has to go through the collecting phase and if there are 22,000 repos that go reset and there collections finished instantly (since the commits exist) it would take around 6 hours for all the repos to get back through to success and this is not even considering the time to add all the tasks to the queue, and the time to clone.

[IsaacMilarky]

I do think it matters, because although it is fast to clone, each repo still has to go through the collecting phase and if there are 22,000 repos that go reset and there collections finished instantly (since the commits exist) it would take around 6 hours for all the repos to get back through to success and this is not even considering the time to add all the tasks to the queue, and the time to clone.

The only issue with this is that it sets all the repos to new but otherwise it doesn't really change the underlying behavior of facade. I do agree that people could see that as a problem though since people want to prioritize the newly added repos.

[ABrain7710]

I approved because it does solve the problem. I'm just a bit apprehensive about doing this without conditions because it may not do what people would expect it to do when it is run multiple times

[IsaacMilarky]

I approved because it does solve the problem. I'm just a bit apprehensive about doing this without conditions because it may not do what people would expect it to do when it is run multiple times

I see the concern but I think any issue this alembic script might cause would be more of an inconvenience caused by not knowing facade repos are all being deleted. If we want to we can add a confirm prompt so that people are absolutely aware that their facade directories are being deleted. If we want to we can also mark the old repos or something so that they aren't re-collected as new.