Indexing and distribution redesign #243
Description
The problem
We're not scalable. Full stop.
The current design of our organization is not scalable to the increasing popularity, needs and update frequency. The system has been failing intermittently (#208) since almost the beginning, but it was good enough.
Now it's not.
The proposal
We need to decentralize this operation. Keep liability and maintenance at minimum. As of writing this post, BSData has 119 repositories. There already was a proposal to deal with the issue: #112 - it was done at 49 repositories.
This means that datafiles should not be BSData-org exclusive. We need to allow indexing of third-party repositories as well. So that a person can setup their own repository and in some semi-automated way add it to the index. This is one part.
The other is that #112 needs to be solved - and the recipe is easier than ever thanks to GitHub Actions. BSData will provide an action that will do all necessary operations so that the BSData indexer only needs to read some manifest file of that repository/release. Each repository will run action by itself, thus offloading the indexer.
The work
Of course, being a volunteer-run entity, there's close to no momentum. I'd love to gain some support from community in refining and implementing this proposal.
I'll be slowly working on that myself, if anyone is wanting to contribute, please do.