-
Notifications
You must be signed in to change notification settings - Fork 319
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add standalone CPEx scraper #3622
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
Codecov ReportAll modified lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #3622 +/- ##
=======================================
Coverage 53.24% 53.24%
=======================================
Files 271 271
Lines 5862 5862
Branches 1396 1396
=======================================
Hits 3121 3121
Misses 2741 2741 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tested and LGTM!
This doesn't
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some changes
import fs from 'fs'; | ||
import path from 'path'; | ||
|
||
import env from '../env.json'; | ||
|
||
// Configure this! | ||
const term = '2310'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Need to set this to 2320
* chore(deps): update dependency postcss to v8.4.31 [security] (#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (#3622) * chore(cpex): set term to next term instead of current (#3631) * Change MPE to CPE and Open MPE (#3623) * Fix serverless function not working (#3623) * Change modules to courses (#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com>
* chore(deps): update dependency postcss to v8.4.31 [security] (#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (#3622) * chore(cpex): set term to next term instead of current (#3631) * Change MPE to CPE and Open MPE (#3623) * Fix serverless function not working (#3623) * Change modules to courses (#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (#3623) * Change package.json to use openssl legacy (#3623) Change package.json to set env variable(#3623) * Remove nodeoptions from package.json and change mpe api link (#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com>
…fications#3632) * chore(deps): update dependency postcss to v8.4.31 [security] (nusmodifications#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (nusmodifications#3622) * chore(cpex): set term to next term instead of current (nusmodifications#3631) * Change MPE to CPE and Open MPE (nusmodifications#3623) * Fix serverless function not working (nusmodifications#3623) * Change modules to courses (nusmodifications#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (nusmodifications#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com>
* chore(deps): update dependency postcss to v8.4.31 [security] (nusmodifications#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (nusmodifications#3622) * chore(cpex): set term to next term instead of current (nusmodifications#3631) * Change MPE to CPE and Open MPE (nusmodifications#3623) * Fix serverless function not working (nusmodifications#3623) * Change modules to courses (nusmodifications#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (nusmodifications#3623) * Change package.json to use openssl legacy (nusmodifications#3623) Change package.json to set env variable(nusmodifications#3623) * Remove nodeoptions from package.json and change mpe api link (nusmodifications#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com>
…fications#3632) * chore(deps): update dependency postcss to v8.4.31 [security] (nusmodifications#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (nusmodifications#3622) * chore(cpex): set term to next term instead of current (nusmodifications#3631) * Change MPE to CPE and Open MPE (nusmodifications#3623) * Fix serverless function not working (nusmodifications#3623) * Change modules to courses (nusmodifications#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (nusmodifications#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com>
* chore(deps): update dependency postcss to v8.4.31 [security] (nusmodifications#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (nusmodifications#3622) * chore(cpex): set term to next term instead of current (nusmodifications#3631) * Change MPE to CPE and Open MPE (nusmodifications#3623) * Fix serverless function not working (nusmodifications#3623) * Change modules to courses (nusmodifications#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (nusmodifications#3623) * Change package.json to use openssl legacy (nusmodifications#3623) Change package.json to set env variable(nusmodifications#3623) * Remove nodeoptions from package.json and change mpe api link (nusmodifications#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com>
* chore(deps): update dependency postcss to v8.4.31 [security] (#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (#3622) * chore(cpex): set term to next term instead of current (#3631) * Change MPE to CPE and Open MPE (#3623) * Fix serverless function not working (#3623) * Change modules to courses (#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com>
* chore(deps): update dependency postcss to v8.4.31 [security] (#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (#3622) * chore(cpex): set term to next term instead of current (#3631) * Change MPE to CPE and Open MPE (#3623) * Fix serverless function not working (#3623) * Change modules to courses (#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (#3623) * Change package.json to use openssl legacy (#3623) Change package.json to set env variable(#3623) * Remove nodeoptions from package.json and change mpe api link (#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com>
* Change MPE to CPE and Open MPE (#3623) (#3627) Prepare for CPEx open * Fix serverless function error 500 in vercel (#3623) (#3628) * Fix serverless function not working (#3623) * Change modules to courses (#3623) (#3629) * Change modules to courses (#3623) * Change inS1 and inS2 MPE to CPEx and fix private key parsing (#3632) * chore(deps): update dependency postcss to v8.4.31 [security] (#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (#3622) * chore(cpex): set term to next term instead of current (#3631) * Change MPE to CPE and Open MPE (#3623) * Fix serverless function not working (#3623) * Change modules to courses (#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com> * Update nus-auth.ts (#3623) Remove previous changes * Try to fix openssl legacy error (#3623) * chore(deps): update dependency postcss to v8.4.31 [security] (#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (#3622) * chore(cpex): set term to next term instead of current (#3631) * Change MPE to CPE and Open MPE (#3623) * Fix serverless function not working (#3623) * Change modules to courses (#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (#3623) * Change package.json to use openssl legacy (#3623) Change package.json to set env variable(#3623) * Remove nodeoptions from package.json and change mpe api link (#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com> * Update ModuleForm.tsx to use CPEx (#3623) * Update FAQ to point to CPEx FAQ (#3623) * Regenerate yarn lockfile using yarn classic v1 --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com>
* chore(deps): update dependency postcss to v8.4.31 [security] (nusmodifications#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (nusmodifications#3622) * chore(cpex): set term to next term instead of current (nusmodifications#3631) * Change MPE to CPE and Open MPE (nusmodifications#3623) * Fix serverless function not working (nusmodifications#3623) * Change modules to courses (nusmodifications#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (nusmodifications#3623) * Change package.json to use openssl legacy (nusmodifications#3623) Change package.json to set env variable(nusmodifications#3623) * Remove nodeoptions from package.json and change mpe api link (nusmodifications#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com>
Context
This PR adds the standalone MPE scraper I've started using since MPE from the last semester.
Why is this needed?
Prior to 2310 MPE, the MPE scraper was coupled with the NUSMods scraper's acad year.
Meaning, in AY2022/2023 S1, when the NUSMods scraper is set to AY2022/2023 S1, we can scrape MPE data for AY2022/2023 S2, which works fine.
However, in AY2022/2023 S2, when the NUSMods scraper is set to AY2022/2023 S2, we cannot scrape MPE data for AY2023/2024 S1, because it is a different acad year. We could just run a separate instance of the NUSMods scraper for AY2023/2024 yes, but because the module data for AY2023/2024 will not be ready by then, the scraper (where we built in a lot of validation) will error out.
The solution, is hence, to build a very simple, standalone MPE scraper, that is not coupled with the NUSMods scraper.
Note: this is definitely not the cleanest, but it's simple, it works, and it does its job well.