Historical event data

[TyHil]

Right now, all of the event data we scrape from Astra, Mazevo, and The Comet Calendar is for the future. We scrape future data and upload it to MongoDB, deleting the old data. I can see a use case for someone needing this historical event data that were deleting, maybe as some kind of training data or something.

Develop a way to upload new Astra, Mazevo, and The Comet Calendar data without deleting the old data. Use the date as the key to replace matching entries during upload.

In addition, check whether the CourseBook events route allows pulling events from the past (as in like over a year ago). If it does not, update the aggregation that generates the events collection to generate all days, instead of just future days.

[bvaic]

I agree. I think it's a good idea to hold on to historical data to help out potential users of our API. If that turns out to be a rare use case, I still think it's good to have that data for archival reasons.

I'm assuming that this involves altering the schemas of astra, mazevo, and cometCalendar to use the date as the key rather than a mongo ObjectId.

This issue seems to have 4 parts (maybe we should make sub-issues?) , 3 of which are related to api-tools which I'm familiar with. I'd love to help out with that.

[TyHil]

I'm not sure about the schema. I think I remember MongoDB automatically giving uploaded documents an _id field when they didn't have one so maybe you could just set the _id to the date and replace matching documents

[bvaic]

That makes sense. Should we keep the date field or just copy it to _id?

[TyHil]

We should keep it so as not to break existing code that relies on it

[mikehquan19]

@bvaic, Just wait till you get all other prs merged I'll reassign you.

[bvaic]

No worries, thanks.