Calculating and Storing Requirement Statistics #866

PabloRaigoza · 2023-10-23T01:00:22Z

Summary

This pull request calculates and stores the data of most common courses to fulfil a given requirement and its slots. It only stores the top 50 most popular courses because storage limitations on Firestore. In addition, this pull request is going to be become important for course recommendations based on most popular courses to fulfill a requirement.

I also cleaned up the code for 'scripts/firebase-config.ts' to be a little more consistant as well as add a new npm command to run most if not all of the cli checks.

Calculating Requirement Statistics
Soring Requirement Statistics

Testing

I tested this script on the dev server by adding a courses on my account, and the changes did appear in Firestore. There are some users that error when attempting to calculate their requirement statistics, by this is likely due to corrupt data in the Firestore.

requirements - Outputs to console the JSON object

- Storing data in firestore

dti-github-bot · 2023-10-23T01:00:43Z

[diff-counting] Significant lines: 187.

github-actions · 2023-10-23T01:02:06Z

Visit the preview URL for this PR (updated for commit 95acbc1):

https://cornelldti-courseplan-dev--pr866-pablo-fulfillment-st-fgfvlq8w.web.app

_{(expires Thu, 28 Dec 2023 18:50:37 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: 6d4e0437c4559ed895272bbd63991394f1e0e933}

elizabeth-tang

This looks really good Pablo! I just had a few small suggestions about code cleaning and console statements. Excited to get started on "recommendations"!!

scripts/gen-req-full-stats.ts

zachary-kent

Just a couple minor nits!

scripts/gen-req-full-stats.ts

src/requirements/fulfillment-stats.ts

scripts/gen-req-full-stats.ts

src/requirements/fulfillment-stats.ts

…fulfillment-stats

zachary-kent

Looks mostly good, just a few small comments?

zachary-kent · 2023-11-05T16:45:25Z

scripts/gen-req-full-stats.ts

+      );
+    } catch {
+      // There was an error computing the fulfillment stats for the user
+      console.log(`${numberOfErrors} : Error computing fulfillment stats for ${doc.id}`);


Suggested change

console.log(`${numberOfErrors} : Error computing fulfillment stats for ${doc.id}`);

console.error(`${numberOfErrors} : Error computing fulfillment stats for ${doc.id}`);

zachary-kent · 2023-11-07T22:46:00Z

scripts/gen-req-full-stats.ts

+  await semQuerySnapshot.forEach(async doc => {
+    // obtain the user's semesters, onboarding data, etc...
+    const semesters = (await doc.data()).semesters ?? {};
+    const onboardingData = (await onboardingDataCollection.doc(doc.id).get()).data() ?? {};
+    const toggleableRequirementChoices =
+      (await toggleableRequirementChoicesCollection.doc(doc.id).get()).data() ?? {};
+    const overriddenFulfillmentChoices =
+      (await overriddenFulfillmentChoicesCollection.doc(doc.id).get()).data() ?? {};
+
+    // Attempt to compute the fulfillment stats for the user
+    try {
+      // use createAppOnboardingData to convert the onboarding data to the format used by the frontend
+      const newOnboardingData = await createAppOnboardingData(onboardingData);
+
+      // compute the fulfillment stats
+      const res = await computeGroupedRequirementFulfillmentReports(
+        semesters,
+        newOnboardingData,
+        toggleableRequirementChoices,
+        overriddenFulfillmentChoices
+      );
+
+      await computeFulfillmentStats(
+        res.groupedRequirementFulfillmentReport,
+        idRequirementFrequency
+      );
+    } catch {
+      // There was an error computing the fulfillment stats for the user
+      console.log(`${numberOfErrors} : Error computing fulfillment stats for ${doc.id}`);
+      numberOfErrors += 1;
+    }
+  });


Suggested change

await semQuerySnapshot.forEach(async doc => {

// obtain the user's semesters, onboarding data, etc...

const semesters = (await doc.data()).semesters ?? {};

const onboardingData = (await onboardingDataCollection.doc(doc.id).get()).data() ?? {};

const toggleableRequirementChoices =

(await toggleableRequirementChoicesCollection.doc(doc.id).get()).data() ?? {};

const overriddenFulfillmentChoices =

(await overriddenFulfillmentChoicesCollection.doc(doc.id).get()).data() ?? {};

// Attempt to compute the fulfillment stats for the user

try {

// use createAppOnboardingData to convert the onboarding data to the format used by the frontend

const newOnboardingData = await createAppOnboardingData(onboardingData);

// compute the fulfillment stats

const res = await computeGroupedRequirementFulfillmentReports(

semesters,

newOnboardingData,

toggleableRequirementChoices,

overriddenFulfillmentChoices

);

await computeFulfillmentStats(

res.groupedRequirementFulfillmentReport,

idRequirementFrequency

);

} catch {

// There was an error computing the fulfillment stats for the user

console.log(`${numberOfErrors} : Error computing fulfillment stats for ${doc.id}`);

numberOfErrors += 1;

}

});

await Promise.all(semQuerySnapshot.map(async doc => {

// obtain the user's semesters, onboarding data, etc...

const semesters = (await doc.data()).semesters ?? {};

const onboardingData = (await onboardingDataCollection.doc(doc.id).get()).data() ?? {};

const toggleableRequirementChoices =

(await toggleableRequirementChoicesCollection.doc(doc.id).get()).data() ?? {};

const overriddenFulfillmentChoices =

(await overriddenFulfillmentChoicesCollection.doc(doc.id).get()).data() ?? {};

// Attempt to compute the fulfillment stats for the user

try {

// use createAppOnboardingData to convert the onboarding data to the format used by the frontend

const newOnboardingData = await createAppOnboardingData(onboardingData);

// compute the fulfillment stats

const res = await computeGroupedRequirementFulfillmentReports(

semesters,

newOnboardingData,

toggleableRequirementChoices,

overriddenFulfillmentChoices

);

await computeFulfillmentStats(

res.groupedRequirementFulfillmentReport,

idRequirementFrequency

);

} catch {

// There was an error computing the fulfillment stats for the user

console.log(`${numberOfErrors} : Error computing fulfillment stats for ${doc.id}`);

numberOfErrors += 1;

}

}));

zachary-kent · 2023-12-01T21:56:29Z

scripts/gen-req-full-stats.ts

+      const newSlot = new Map<number, number>();
+      const sorted = [...slot.entries()].sort((a, b) => b[1] - a[1]);
+
+      const numberOfCourses = sorted.length > 50 ? 50 : sorted.length;


Suggested change

const numberOfCourses = sorted.length > 50 ? 50 : sorted.length;

const numberOfCourses = Math.min(sorted.length, 50);

zachary-kent · 2023-12-01T21:59:09Z

scripts/gen-req-full-stats.ts

+ * @param _callback is a function that is called after the fulfillment stats have been computed
+ * @throws Error when computeGroupedRequirementFulfillmentReports fails to compute the fulfillment stats
+ */
+async function computeRequirementFullfillmentStatistics(_callback) {


Why do we need this _callback argument?

zachary-kent · 2023-12-01T22:00:52Z

src/requirements/fulfillment-stats.ts

+        for (let j = 0; j < currentCourseSlot.length; j += 1) {
+          const currentCourseId = currentCourseSlot[j].courseId;
+          const pastFreq = currentRequirementSlotFreq.get(currentCourseId) ?? 0;
+          currentRequirementSlotFreq.set(currentCourseId, pastFreq + 1);
+        }


Can you convert this into a for each loop?

PabloRaigoza added 7 commits September 21, 2023 17:56

Adding new file in scripts and debugging credentials

ca6a39d

Implementing the script to determine course frequencies to fulfill given

f11b11e

requirements - Outputs to console the JSON object

Fixing formatting and linting errors

06a026f

Implmenting gen-req-full-stats

d06218c

- Storing data in firestore

Ran format

6aab6fe

Adding documention and removing console.log

ccbfee3

Ran format

7b5ba5c

PabloRaigoza requested a review from a team as a code owner October 23, 2023 01:00

elizabeth-tang requested changes Oct 23, 2023

View reviewed changes

scripts/gen-req-full-stats.ts Outdated Show resolved Hide resolved

scripts/gen-req-full-stats.ts Outdated Show resolved Hide resolved

scripts/gen-req-full-stats.ts Outdated Show resolved Hide resolved

scripts/gen-req-full-stats.ts Outdated Show resolved Hide resolved

zachary-kent requested changes Oct 23, 2023

View reviewed changes

PabloRaigoza added 2 commits November 5, 2023 09:49

Merge branch 'main' of github.com:cornell-dti/course-plan into pablo/…

4f326fd

…fulfillment-stats

Cleaning code and replacing backslash with doc id

a8a5630

PabloRaigoza mentioned this pull request Nov 13, 2023

Implementing Frontend Changes for Recomendations #873

Open

1 task

Resolving all promises before callback

95acbc1

PabloRaigoza requested review from elizabeth-tang and zachary-kent November 28, 2023 18:52

zachary-kent requested changes Dec 1, 2023

View reviewed changes

PabloRaigoza mentioned this pull request Dec 4, 2023

Load Requirement Ranking for Frontend #876

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Calculating and Storing Requirement Statistics #866

Calculating and Storing Requirement Statistics #866

PabloRaigoza commented Oct 23, 2023

dti-github-bot commented Oct 23, 2023 •

edited

Loading

github-actions bot commented Oct 23, 2023 •

edited

Loading

elizabeth-tang left a comment

zachary-kent left a comment

zachary-kent left a comment

zachary-kent Nov 5, 2023

zachary-kent Nov 7, 2023

zachary-kent Dec 1, 2023

zachary-kent Dec 1, 2023

zachary-kent Dec 1, 2023

	console.log(`${numberOfErrors} : Error computing fulfillment stats for ${doc.id}`);
	console.error(`${numberOfErrors} : Error computing fulfillment stats for ${doc.id}`);

	const numberOfCourses = sorted.length > 50 ? 50 : sorted.length;
	const numberOfCourses = Math.min(sorted.length, 50);

Calculating and Storing Requirement Statistics #866

Are you sure you want to change the base?

Calculating and Storing Requirement Statistics #866

Conversation

PabloRaigoza commented Oct 23, 2023

Summary

Testing

dti-github-bot commented Oct 23, 2023 • edited Loading

github-actions bot commented Oct 23, 2023 • edited Loading

elizabeth-tang left a comment

Choose a reason for hiding this comment

zachary-kent left a comment

Choose a reason for hiding this comment

zachary-kent left a comment

Choose a reason for hiding this comment

zachary-kent Nov 5, 2023

Choose a reason for hiding this comment

zachary-kent Nov 7, 2023

Choose a reason for hiding this comment

zachary-kent Dec 1, 2023

Choose a reason for hiding this comment

zachary-kent Dec 1, 2023

Choose a reason for hiding this comment

zachary-kent Dec 1, 2023

Choose a reason for hiding this comment

dti-github-bot commented Oct 23, 2023 •

edited

Loading

github-actions bot commented Oct 23, 2023 •

edited

Loading