Add standalone CPEx scraper #3622

chrisgzf · 2023-10-01T11:15:05Z

Context

This PR adds the standalone MPE scraper I've started using since MPE from the last semester.

Why is this needed?

Prior to 2310 MPE, the MPE scraper was coupled with the NUSMods scraper's acad year.

Meaning, in AY2022/2023 S1, when the NUSMods scraper is set to AY2022/2023 S1, we can scrape MPE data for AY2022/2023 S2, which works fine.

However, in AY2022/2023 S2, when the NUSMods scraper is set to AY2022/2023 S2, we cannot scrape MPE data for AY2023/2024 S1, because it is a different acad year. We could just run a separate instance of the NUSMods scraper for AY2023/2024 yes, but because the module data for AY2023/2024 will not be ready by then, the scraper (where we built in a lot of validation) will error out.

The solution, is hence, to build a very simple, standalone MPE scraper, that is not coupled with the NUSMods scraper.

Note: this is definitely not the cleanest, but it's simple, it works, and it does its job well.

vercel · 2023-10-01T11:15:12Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
nusmods-export	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 8, 2023 5:32am
nusmods-website	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 8, 2023 5:32am

codecov · 2023-10-01T11:18:50Z

Codecov Report

All modified lines are covered by tests ✅

Comparison is base (7312bf0) 53.24% compared to head (c63f1f5) 53.24%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #3622   +/-   ##
=======================================
  Coverage   53.24%   53.24%           
=======================================
  Files         271      271           
  Lines        5862     5862           
  Branches     1396     1396           
=======================================
  Hits         3121     3121           
  Misses       2741     2741

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ravern

Tested and LGTM!

ravern · 2023-10-08T04:59:53Z

This doesn't

Remove the old MPE scraper, which should still produce mpeModules.json in the semester-level directory
Modify the frontend to use the new cpexModules.json file

chrisgzf

Some changes

chrisgzf · 2023-10-08T08:36:36Z

scrapers/cpex-scraper/src/index.ts

 import fs from 'fs';
 import path from 'path';

+import env from '../env.json';
+
 // Configure this!
 const term = '2310';


Need to set this to 2320

* chore(deps): update dependency postcss to v8.4.31 [security] (#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (#3622) * chore(cpex): set term to next term instead of current (#3631) * Change MPE to CPE and Open MPE (#3623) * Fix serverless function not working (#3623) * Change modules to courses (#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com>

* chore(deps): update dependency postcss to v8.4.31 [security] (#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (#3622) * chore(cpex): set term to next term instead of current (#3631) * Change MPE to CPE and Open MPE (#3623) * Fix serverless function not working (#3623) * Change modules to courses (#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (#3623) * Change package.json to use openssl legacy (#3623) Change package.json to set env variable(#3623) * Remove nodeoptions from package.json and change mpe api link (#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com>

…fications#3632) * chore(deps): update dependency postcss to v8.4.31 [security] (nusmodifications#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (nusmodifications#3622) * chore(cpex): set term to next term instead of current (nusmodifications#3631) * Change MPE to CPE and Open MPE (nusmodifications#3623) * Fix serverless function not working (nusmodifications#3623) * Change modules to courses (nusmodifications#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (nusmodifications#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com>

* chore(deps): update dependency postcss to v8.4.31 [security] (nusmodifications#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (nusmodifications#3622) * chore(cpex): set term to next term instead of current (nusmodifications#3631) * Change MPE to CPE and Open MPE (nusmodifications#3623) * Fix serverless function not working (nusmodifications#3623) * Change modules to courses (nusmodifications#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (nusmodifications#3623) * Change package.json to use openssl legacy (nusmodifications#3623) Change package.json to set env variable(nusmodifications#3623) * Remove nodeoptions from package.json and change mpe api link (nusmodifications#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com>

…fications#3632) * chore(deps): update dependency postcss to v8.4.31 [security] (nusmodifications#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (nusmodifications#3622) * chore(cpex): set term to next term instead of current (nusmodifications#3631) * Change MPE to CPE and Open MPE (nusmodifications#3623) * Fix serverless function not working (nusmodifications#3623) * Change modules to courses (nusmodifications#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (nusmodifications#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com>

* chore(deps): update dependency postcss to v8.4.31 [security] (nusmodifications#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (nusmodifications#3622) * chore(cpex): set term to next term instead of current (nusmodifications#3631) * Change MPE to CPE and Open MPE (nusmodifications#3623) * Fix serverless function not working (nusmodifications#3623) * Change modules to courses (nusmodifications#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (nusmodifications#3623) * Change package.json to use openssl legacy (nusmodifications#3623) Change package.json to set env variable(nusmodifications#3623) * Remove nodeoptions from package.json and change mpe api link (nusmodifications#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com>

* chore(deps): update dependency postcss to v8.4.31 [security] (#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (#3622) * chore(cpex): set term to next term instead of current (#3631) * Change MPE to CPE and Open MPE (#3623) * Fix serverless function not working (#3623) * Change modules to courses (#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com>

* chore(deps): update dependency postcss to v8.4.31 [security] (#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (#3622) * chore(cpex): set term to next term instead of current (#3631) * Change MPE to CPE and Open MPE (#3623) * Fix serverless function not working (#3623) * Change modules to courses (#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (#3623) * Change package.json to use openssl legacy (#3623) Change package.json to set env variable(#3623) * Remove nodeoptions from package.json and change mpe api link (#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com>

* Change MPE to CPE and Open MPE (#3623) (#3627) Prepare for CPEx open * Fix serverless function error 500 in vercel (#3623) (#3628) * Fix serverless function not working (#3623) * Change modules to courses (#3623) (#3629) * Change modules to courses (#3623) * Change inS1 and inS2 MPE to CPEx and fix private key parsing (#3632) * chore(deps): update dependency postcss to v8.4.31 [security] (#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (#3622) * chore(cpex): set term to next term instead of current (#3631) * Change MPE to CPE and Open MPE (#3623) * Fix serverless function not working (#3623) * Change modules to courses (#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com> * Update nus-auth.ts (#3623) Remove previous changes * Try to fix openssl legacy error (#3623) * chore(deps): update dependency postcss to v8.4.31 [security] (#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (#3622) * chore(cpex): set term to next term instead of current (#3631) * Change MPE to CPE and Open MPE (#3623) * Fix serverless function not working (#3623) * Change modules to courses (#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (#3623) * Change package.json to use openssl legacy (#3623) Change package.json to set env variable(#3623) * Remove nodeoptions from package.json and change mpe api link (#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com> * Update ModuleForm.tsx to use CPEx (#3623) * Update FAQ to point to CPEx FAQ (#3623) * Regenerate yarn lockfile using yarn classic v1 --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com>

* chore(deps): update dependency postcss to v8.4.31 [security] (nusmodifications#3630) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * feat(cpex): add standalone scraper (nusmodifications#3622) * chore(cpex): set term to next term instead of current (nusmodifications#3631) * Change MPE to CPE and Open MPE (nusmodifications#3623) * Fix serverless function not working (nusmodifications#3623) * Change modules to courses (nusmodifications#3623) * Change inS1 and inS2 from mpe to cpex and fix private key parsing (nusmodifications#3623) * Change package.json to use openssl legacy (nusmodifications#3623) Change package.json to set env variable(nusmodifications#3623) * Remove nodeoptions from package.json and change mpe api link (nusmodifications#3623) --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Christopher Goh <chrisgzf@gmail.com> Co-authored-by: Ravern Koh <ravernkoh@gmail.com>

chrisgzf added 8 commits October 1, 2023 19:01

Add standalone MPE scraper

94c74da

Save files to data folder

82ab015

Fix package.json commands

5566643

Add pm2 config for MPE scraper

9b89312

Filter hidden modules

a4d4300

Add number of modules threshold check

25448d1

Filter duplicate modules

42403d3

Update threshold after dupe fix

394efa4

chrisgzf requested a review from a team October 1, 2023 11:15

This was referenced Oct 1, 2023

CPEx 2320 - Set up CPE scraper #3624

Closed

CPEx Tech Debt - Remove generation of MPE modules from NUSMods scraper #3625

Open

ravern added 2 commits October 8, 2023 00:49

Rename "MPE" references to "CPEx"

cb9b17a

Rename mpe-modules to cpex-modules dir

743a0b7

vercel bot deployed to Preview – nusmods-export October 7, 2023 16:51 View deployment

vercel bot deployed to Preview – nusmods-website October 7, 2023 16:53 View deployment

Rename remaining references to "CPEx"

fc2fcb1

vercel bot deployed to Preview – nusmods-export October 7, 2023 16:58 View deployment

vercel bot deployed to Preview – nusmods-website October 7, 2023 17:00 View deployment

ravern approved these changes Oct 7, 2023

View reviewed changes

ravern changed the title ~~Add standalone MPE scraper~~ Add standalone CPEx scraper Oct 8, 2023

Add cpexModules.json to rsync exclude

f21830c

vercel bot deployed to Preview – nusmods-export October 8, 2023 05:05 View deployment

vercel bot deployed to Preview – nusmods-website October 8, 2023 05:07 View deployment

Merge branch 'master' into standalone-mpe-scraper

c63f1f5

vercel bot deployed to Preview – nusmods-export October 8, 2023 05:29 View deployment

vercel bot deployed to Preview – nusmods-website October 8, 2023 05:32 View deployment

ravern merged commit 23919f7 into master Oct 8, 2023
6 checks passed

ravern deleted the standalone-mpe-scraper branch October 8, 2023 07:12

chrisgzf commented Oct 8, 2023

View reviewed changes

ravern mentioned this pull request Oct 8, 2023

chore(cpex): set term to next term instead of current #3631

Merged

alissayarmantho pushed a commit to alissayarmantho/nusmods that referenced this pull request Oct 8, 2023

feat(cpex): add standalone scraper (nusmodifications#3622)

a2e555d

alissayarmantho pushed a commit to alissayarmantho/nusmods that referenced this pull request Oct 14, 2023

feat(cpex): add standalone scraper (nusmodifications#3622)

ec1e198

alissayarmantho pushed a commit to alissayarmantho/nusmods that referenced this pull request Oct 14, 2023

feat(cpex): add standalone scraper (nusmodifications#3622)

bda9763

alissayarmantho pushed a commit to alissayarmantho/nusmods that referenced this pull request Oct 14, 2023

feat(cpex): add standalone scraper (nusmodifications#3622)

576cdc4

alissayarmantho pushed a commit to alissayarmantho/nusmods that referenced this pull request Oct 21, 2023

feat(cpex): add standalone scraper (nusmodifications#3622)

8c2f384

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add standalone CPEx scraper #3622

Add standalone CPEx scraper #3622

chrisgzf commented Oct 1, 2023

vercel bot commented Oct 1, 2023 •

edited

Loading

codecov bot commented Oct 1, 2023 •

edited

Loading

ravern left a comment

ravern commented Oct 8, 2023

chrisgzf left a comment

chrisgzf Oct 8, 2023

Add standalone CPEx scraper #3622

Add standalone CPEx scraper #3622

Conversation

chrisgzf commented Oct 1, 2023

Context

Why is this needed?

vercel bot commented Oct 1, 2023 • edited Loading

codecov bot commented Oct 1, 2023 • edited Loading

Codecov Report

ravern left a comment

Choose a reason for hiding this comment

ravern commented Oct 8, 2023

chrisgzf left a comment

Choose a reason for hiding this comment

chrisgzf Oct 8, 2023

Choose a reason for hiding this comment

vercel bot commented Oct 1, 2023 •

edited

Loading

codecov bot commented Oct 1, 2023 •

edited

Loading