972: Replace unnecessary data query

[rosado]

What type of PR is this? (check all applicable)

• Refactor
• Bug Fix
• Optimization

Description

Whole set of entities was fetched when only the count was needed.

Related Tickets & Documents

• Closes Erroneous entity count in dataset task list. #972

QA Instructions, Screenshots, Recordings

Added/updated tests?

• Yes

QA sign off

• Code has been checked and approved
• Design has been checked and approved
• Product and business logic has been checked and proved

Summary by CodeRabbit

• New Features

  • Introduced a new service for accurately retrieving entity counts associated with organisations.

• Optimisations

  • Enhanced the efficiency of filtering items with associated issues.
  • Updated task preparation workflows to ensure more consistent handling of entity counts.

• Tests

  • Expanded test scenarios to validate various entity count conditions for improved reliability.

[coderabbitai]

Walkthrough

This pull request introduces a new utility function, fetchEntityCount, which retrieves the count of entities via a SQL query. The update replaces the older fetchEntitiesCount and refines logic in the filterOutEntitiesWithoutIssues function by using a Set for improved efficiency. In addition, middleware in the dataset task list has been updated to use the new fetchEntityCount for determining the entity count, and related tests have been modified to include an entityCount property on the request object.

Changes

File(s)	Change Summary
src/middleware/common.middleware.js src/middleware/dataview.middleware.js	Added the fetchEntityCount function to retrieve SQL entity counts; removed fetchEntitiesCount from dataview.middleware.js; improved the filtering logic in common.middleware.js by utilising a Set for efficiency.
src/middleware/datasetTaskList.middleware.js	Replaced the processEntitiesMiddlewares with fetchEntityCount; updated the parameter documentation of entityOutOfBoundsMessage; revised prepareTasks to extract entityCount.count instead of relying on an array’s length.
test/unit/middleware/datasetTaskList.middleware.test.js	Updated tests to include the entityCount property on the request object with various values to validate the new logic in prepareTasks.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant DTM as DatasetTask Middleware
    participant CM as Common Middleware (fetchEntityCount)
    participant DB as Database

    Client->>DTM: Send request with org info
    DTM->>CM: Call fetchEntityCount
    CM->>DB: Execute SQL query (SELECT COUNT(*))
    DB-->>CM: Return entityCount
    CM-->>DTM: Return fetched entityCount
    DTM->>DTM: Use req.entityCount.count in prepareTasks
    DTM->>Client: Process and respond to request

Loading

Assessment against linked issues

Objective	Addressed	Explanation
Correct the entity count mechanism in dataset task list (#972)	✅

Possibly related PRs

• 682 dataset task list fetch data from all active resources #717: The changes in the main PR are related to the retrieved PR as both introduce the fetchEntityCount function and modify how entity counts are handled in the middleware, specifically affecting the same functions and logic in datasetTaskList.middleware.js.
• express error handling #657: The changes in the main PR are related to the introduction and modification of the fetchEntityCount function, which is also referenced in the retrieved PR as it replaces the fetchEntitiesCount function, indicating a direct connection in functionality.
• refactor prepareEntity() #986: The changes in the main PR are related to the introduction and usage of the fetchEntityCount function, which is also referenced in the retrieved PR, indicating a direct connection in how entity counts are managed across the middleware.

Suggested labels

enhancement, tech debt

Suggested reviewers

• GeorgeGoodall-GovUk
• DilwoarH

Poem

I'm a rabbit with a hop in my stride,
New code in place with counts bona fide.
SQL queries now count, without excess load,
Efficient filtering - a lighter code road.
Hopping through changes with a witty cheer,
Celebrating smooth functions, far and near!
🐰✨

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 9dc95b1 and c0aeaa5.

📒 Files selected for processing (3)

• src/middleware/common.middleware.js (2 hunks)
• src/middleware/datasetTaskList.middleware.js (5 hunks)
• test/unit/middleware/datasetTaskList.middleware.test.js (3 hunks)

🚧 Files skipped from review as they are similar to previous changes (1)

• test/unit/middleware/datasetTaskList.middleware.test.js

🧰 Additional context used

🧬 Code Definitions (1)

src/middleware/common.middleware.js (4)

src/middleware/datasetOverview.middleware.js (2)

• fetchEntityCount (132-141)
• fetchEntityCount (132-141)

src/middleware/middleware.builders.js (8)

• fetchOne (273-275)
• FetchOptions (28-37)
• FetchOptions (28-37)
• result (93-93)
• result (121-121)
• result (159-159)
• result (200-200)
• result (228-228)

src/middleware/entityIssueDetails.middleware.js (1)

• req (87-87)

src/utils/pagination.js (1)

• result (39-43)

🔇 Additional comments (7)

src/middleware/common.middleware.js (2)

305-311: Good addition of the fetchEntityCount function.

This new function efficiently retrieves just the count of entities rather than fetching all entities, which aligns with the PR objective of eliminating unnecessary data queries.

---

377-389: Great optimisation using Set for entity filtering.

The refactoring of this function to use a Set for entity lookups is an excellent performance improvement. Set lookups are O(1) rather than the O(n) operations that would be needed with array methods like .filter() and .some().

src/middleware/datasetTaskList.middleware.js (5)

17-28: Good update to imports.

Replacing the import of the removed processEntitiesMiddlewares with the new fetchEntityCount function is appropriate for the optimization being implemented.

---

90-90: Updated documentation to reflect new data structure.

Correctly updated the JSDoc to indicate that req.entityCount now represents the total entity count as an object with a count field.

---

107-107: Good refactoring to use the more efficient entity count.

Updating the destructuring to extract entityCount directly from req instead of entities aligns with the new approach.

---

122-122: Correctly implemented the new data structure.

Using entityCount.count directly instead of counting entities after fetching them is more efficient.

---

221-221: Excellent middleware sequence update.

Replacing ...processEntitiesMiddlewares with the focused fetchEntityCount function completes the optimization goal of only fetching what's needed.

✨ Finishing Touches

• 📝 Generate Docstrings

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai plan to trigger planning for file edits and PR creation.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	66.13%	5834 / 8822
🔵	Statements	66.13%	5834 / 8822
🔵	Functions	64.73%	246 / 380
🔵	Branches	79.13%	751 / 949

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
src/middleware/common.middleware.js	69.42%	92.85%	39.62%	69.42%	29-38, 42, 58-59, 61-62, 74-76, 80, 96-103, 162-173, 179-201, 299, 307, 316-318, 348-350, 398-412, 452-456, 461-464, 476-480, 508-522, 529-540, 546-554, 558-574, 643-645, 718-725, 739-794, 825-826, 829-832, 835-848, 885-888
src/middleware/datasetTaskList.middleware.js	93.51%	80.95%	66.66%	93.51%	127-128, 155-164
src/middleware/dataview.middleware.js	100%	100%	60%	100%

Generated in workflow #1031 for commit c0aeaa5 by the Vitest Coverage Report Action