Migrate file size/count queries to use ICAT v5 properties for investigation/dataset entities #1499

jounaidr · 2023-01-17T18:31:59Z

Description

With ICAT V5.0.0 the underlying file sizes and counts for dataset and investigation entities are included as properties. Previously when a file size/count was required for any table/card views, it was calculated on the fly using a query. This PR removes said query methods and where they are used and replaces them with the values directly from the received entitles.

Note that I was not able to test the changes made to dls table and card views, so the changes made to the respective files are assumed to work but should be double checked before this PR is approved. Also, the fetchSize() size buttons that were previously implemented within dls/datasetDetailsPanel.component.tsx and dls/visitDetailsPanel.component.tsx have been removed since the size is no longer calculated using the query methods mentioned. Related e2e tests have been removed as well. Maybe the file count columns should be replaced with the file sizes in dls views?

The formatCountOrSize() method within cellContentRenderers.tsx is now somewhat redundant as the formatBytes() method is now imported directly when used, so this method could possibly be refactored to only format the count (i.e formatCount()). However the method is used elsewhere so I think this is out of scope for this issue.

To Finish PR

As mentioned in review comments, some of datagateway-download downloadCartTable can also be refactored to only use one query per item in the cart. An attempt has been made, with new API to request the entire entities, and to use the new fileSize/fileCount properties within downloadCartTable. Using multiple types for Datafile Datasets and Investigations seems to be working, however as expected the file count method needs changes to identify if the entity is a datafile as count property doesn't exist. see here

Testing instructions

Check each page where a file size or count has been refactors still correctly displays the respective info.

Agile board tracking

Closes #1457

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

- add fileSize and fileCount props to Investigation interface - migrate investigation sizes in isisInvestigationsCardView.component.tsx Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

- add fileSize and fileCount props to Dataset Datafile interfaces, set all to possibly undefined - add fileCount prop to test Datafile - make download button entity size nullish in isisInvestigationsCardView.component.tsx Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

- migrate file size in investigationSearchTable.component.tsx - add TODO comment in cellContentRenderers.tsx formatCountOrSize() function Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

…size as dls hasn't upgraded to ICAT v5 yet Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

…ponent.tsx Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

…ent.tsx Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

- migrate dataset size in dls datasetDetailsPanel.component.tsx - remove useDatasetSize query from datasets.tsx Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

…note: untested) Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

- fix common package unit tests - remove fetch() size button from dls pages Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

jounaidr · 2023-01-24T22:57:01Z

packages/datagateway-common/src/detailsPanels/dls/visitDetailsPanel.component.tsx

@@ -205,23 +195,7 @@ const VisitDetailsPanel = (
              {t('investigations.details.size')}
            </Typography>
            <Typography>
-              <b>


fetch() size button removed as I believe its redundant now, however I haven't test dls pages

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

# Conflicts: # packages/datagateway-dataview/src/views/card/dls/__snapshots__/dlsDatasetsCardView.component.test.tsx.snap # packages/datagateway-dataview/src/views/card/dls/__snapshots__/dlsVisitsCardView.component.test.tsx.snap # packages/datagateway-dataview/src/views/card/isis/isisInvestigationsCardView.component.test.tsx # packages/datagateway-dataview/src/views/landing/isis/isisDatasetLanding.component.test.tsx # packages/datagateway-dataview/src/views/landing/isis/isisInvestigationLanding.component.test.tsx # packages/datagateway-dataview/src/views/table/__snapshots__/datasetTable.component.test.tsx.snap # packages/datagateway-dataview/src/views/table/datasetTable.component.test.tsx # packages/datagateway-dataview/src/views/table/dls/__snapshots__/dlsDatasetsTable.component.test.tsx.snap # packages/datagateway-dataview/src/views/table/dls/__snapshots__/dlsMyDataTable.component.test.tsx.snap # packages/datagateway-dataview/src/views/table/dls/__snapshots__/dlsVisitsTable.component.test.tsx.snap # packages/datagateway-dataview/src/views/table/dls/dlsDatasetsTable.component.test.tsx # packages/datagateway-dataview/src/views/table/investigationTable.component.test.tsx # packages/datagateway-dataview/src/views/table/isis/isisDatasetsTable.component.test.tsx # packages/datagateway-dataview/src/views/table/isis/isisInvestigationsTable.component.test.tsx # packages/datagateway-dataview/src/views/table/isis/isisMyDataTable.component.test.tsx # packages/datagateway-search/src/card/__snapshots__/datasetSearchCardView.component.test.tsx.snap # packages/datagateway-search/src/card/__snapshots__/investigationSearchCardView.component.test.tsx.snap # packages/datagateway-search/src/card/datasetSearchCardView.component.test.tsx # packages/datagateway-search/src/card/investigationSearchCardView.component.test.tsx # packages/datagateway-search/src/table/__snapshots__/datasetSearchTable.component.test.tsx.snap # packages/datagateway-search/src/table/__snapshots__/investigationSearchTable.component.test.tsx.snap # packages/datagateway-search/src/table/datasetSearchTable.component.test.tsx # packages/datagateway-search/src/table/investigationSearchTable.component.test.tsx

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

louise-davies

There's also some refactoring needed in datagateway-download - as downloadCartTable displays the size + file count of each entity in the cart to both display and accumulate the total size + file count. There's some efficiency to be had in only going through and fetching each entity in the cart once, and using the resulting Datafiles/Datasets/Investigations to derive both the sizes and counts - whereas currently we're sending two queries for each entity. lmk if you want to chat with me to explain this better.

Otherwise, just some minor comments and some things pending on knowing what DLS want

louise-davies · 2023-03-23T09:40:20Z

packages/datagateway-common/src/api/investigations.tsx

-
-  return sizes;
-};
-
 export const useInvestigationsDatasetCount = (


We can also remove this function correct? As Investigation already has a fileCount field. Though I suppose that's working out the number of datafiles instead of the number of datasets. I guess we need to ask Diamond what info they'd like to display in the tables - I think originally count was only used because calculating sizes would be too inefficient...

packages/datagateway-common/src/detailsPanels/dls/datasetDetailsPanel.component.tsx

packages/datagateway-common/src/detailsPanels/dls/visitDetailsPanel.component.tsx

louise-davies · 2023-03-23T09:44:14Z

packages/datagateway-common/src/table/cellRenderers/cellContentRenderers.tsx

@@ -32,6 +32,8 @@ export function formatBytes(bytes: number | undefined): string {
 * @param formatAsBytes Whether to format the data as bytes, default is false
 * @returns a string with either Calculating, Unknown or the formatted data
 */
+//TODO: Since filesize is now given as an entity property, this function can be


TODO left in here - should probably just get resolved. However, if we decide that we no longer need to fetch investigation dataset counts then this function could be removed altogether I think?

packages/datagateway-common/src/app.types.tsx

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

…d count Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

louise-davies · 2023-05-17T14:20:54Z

I've addressed the review comments I had previously made and ensured all the tests are passing - we're now just waiting on a decision from Diamond on what they want to show in their views

louise-davies · 2024-01-18T14:19:57Z

Dataset count replaced with investigation filesize

Dataset datafile count replaced with the ICAT 5 equivalent, and add a size column

Waiting confimation from Kirsty and Greg

Also, let users sort by size or count And, remove DLS datafile size filtering

kaperoo

Looks good to me 👍

jounaidr added 27 commits January 17, 2023 16:24

size prop refactored in investigationTable.component.tsx

4a54763

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

investigationTable.component.tsx cleanup

c2fa492

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

Changes:

9083f54

- add fileSize and fileCount props to Investigation interface - migrate investigation sizes in isisInvestigationsCardView.component.tsx Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

migrate file size in isisInvestigationLanding.component.tsx

a265ad6

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

migrate file size in isisInvestigationsTable.component.tsx

a6075a7

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

migrate file size in isisMyDataTable.component.tsx

d533d16

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

changes:

94fbe0f

- migrate file size in investigationSearchTable.component.tsx - add TODO comment in cellContentRenderers.tsx formatCountOrSize() function Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

migrate file size in investigationSearchCardView.component.tsx

693412c

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

remove useInvestigationSizes method

28244bc

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

Add TODO in visitDetailsPanel.component.tsx to migrate investigation …

6bfbb43

…size as dls hasn't upgraded to ICAT v5 yet Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

migrate file size in isisDatasetsCardView.component.tsx

6c47cd4

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

migrate Dataset file size in isisDatasetsTable.component.tsx

a3046be

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

migrate Dataset file size in isisDatasetLanding.component.tsx

1698cc4

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

migrate Dataset file size and file count in datasetSearchCardView.com…

fb8c883

…ponent.tsx Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

migrate Dataset file size and file count in datasetSearchTable.compon…

e08f2e9

…ent.tsx Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

remove useDatasetSizes function from datasets.tsx

639b4e2

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

migrate Dataset Datafile count in datasetTable.component.tsx

1074a28

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

changes (note: untested):

cefc360

- migrate dataset size in dls datasetDetailsPanel.component.tsx - remove useDatasetSize query from datasets.tsx Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

migrate Dataset Datafile count in dlsDatasetsCardView.component.tsx (…

027d1ce

…note: untested) Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

migrate Dataset Datafile count in dlsDatasetsCardView.component.tsx (…

c989c5b

…note: untested) Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

remove useDatasetsDatafileCount query from datasets.tsx api

5d633e9

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

migrate useInvestigationSize in dls visitDetailsPanel.component.tsx

8638b25

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

remove useInvestigationSize query from investigations.tsx api

edaa4f3

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

remove cellContentRenderer from datasetTable.component.tsx

43aa829

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

remove cellContentRenderer from dlsDatasetsTable.component.tsx

c98d403

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

changes:

12d8989

- fix common package unit tests - remove fetch() size button from dls pages Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

jounaidr commented Jan 24, 2023

View reviewed changes

jounaidr added 2 commits January 25, 2023 16:00

fix dataview unit tests

935e250

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

fix serach unit tests

ce1e263

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

jounaidr added 5 commits March 2, 2023 13:38

remove 'and then calculate file size' test cases from e2e

3544654

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

fix failing e2e

3867506

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

fix failing e2e

baa448b

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

fix failing e2e

bf976fb

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

fix failing e2e

524a1a1

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

jounaidr requested a review from louise-davies March 3, 2023 14:51

jounaidr added 3 commits March 13, 2023 15:08

fix all dataview tests

062e536

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

fix all search tests

47889ca

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

louise-davies requested changes Mar 23, 2023

View reviewed changes

jounaidr and others added 9 commits March 23, 2023 13:14

change rowData to respective entity for consistency

a18562b

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

remove fileCount from datafile interface

d58ae8e

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

datagateway download use single query instead of two for both size an…

6bad65b

…d count Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

remove dev code

01b16c1

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

fix types

e51e4fc

Signed-off-by: jounaidr <jounaidruhomaun@googlemail.com>

#1457 - Finish refactoring download cart sizes & counts

15ef30c

#1457 - update unit tests for refactored cart sizes & counts

1ceaec0

#1457 - remove wrong datafile fileCount property

043f469

#1457 - fix size & count mocking in downloadTab

cebcd8b

louise-davies mentioned this pull request Jun 29, 2023

Lucene search UI changes #1401

Open

3 tasks

louise-davies added 3 commits February 5, 2024 13:19

Merge branch 'develop' into feature/use-icat5-file-count-#1457

3b00c83

#1457 - remove calculation of dataset counts at DLS investigation level

814dd54

#1457 - add size column to DLS dataset views

a3da1b6

Also, let users sort by size or count And, remove DLS datafile size filtering

louise-davies requested a review from kaperoo February 6, 2024 09:30

kaperoo approved these changes Feb 6, 2024

View reviewed changes

louise-davies approved these changes Feb 6, 2024

View reviewed changes

louise-davies merged commit f227864 into develop Feb 6, 2024
10 checks passed

louise-davies deleted the feature/use-icat5-file-count-#1457 branch February 6, 2024 15:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate file size/count queries to use ICAT v5 properties for investigation/dataset entities #1499

Migrate file size/count queries to use ICAT v5 properties for investigation/dataset entities #1499

jounaidr commented Jan 17, 2023 •

edited

Loading

jounaidr Jan 24, 2023 •

edited

Loading

louise-davies left a comment

louise-davies Mar 23, 2023

louise-davies Mar 23, 2023

louise-davies commented May 17, 2023

louise-davies commented Jan 18, 2024

kaperoo left a comment

Migrate file size/count queries to use ICAT v5 properties for investigation/dataset entities #1499

Migrate file size/count queries to use ICAT v5 properties for investigation/dataset entities #1499

Conversation

jounaidr commented Jan 17, 2023 • edited Loading

Description

To Finish PR

Testing instructions

Agile board tracking

jounaidr Jan 24, 2023 • edited Loading

Choose a reason for hiding this comment

louise-davies left a comment

Choose a reason for hiding this comment

louise-davies Mar 23, 2023

Choose a reason for hiding this comment

louise-davies Mar 23, 2023

Choose a reason for hiding this comment

louise-davies commented May 17, 2023

louise-davies commented Jan 18, 2024

kaperoo left a comment

Choose a reason for hiding this comment

jounaidr commented Jan 17, 2023 •

edited

Loading

jounaidr Jan 24, 2023 •

edited

Loading