Skip to content

Reduce data output for Diffcal and Reduction#425

Merged
dlcaballero16 merged 1 commit intostagingfrom
reduce-data-size
Jul 30, 2024
Merged

Reduce data output for Diffcal and Reduction#425
dlcaballero16 merged 1 commit intostagingfrom
reduce-data-size

Conversation

@dlcaballero16
Copy link
Collaborator

Description of work

Added flag to not preserve events for DiffCal and Reduction workflows. This prevents file/workspace size from getting too big.

To test

Dev testing

On analysis, make sure you are on the staging branch. Run the diffcal workflow with the following parameters:

Run Number: 59039
Sample: La11B6_NIST_660c_001
Grouping File: Column
Peak Intensity Threshold: 0.01

Click continue. On the Tweak Peak Peek tab, set MaxChiSq to 2000, click Recalculate, and continue on all the way to the Saving tab. Before saving, inspect the output workspaces. You should see that the main output workspace takes up about 3000 MB of space. Complete the save step and verify there that the tar file and diagnostic .nxs file are at least 3 GB.

Run Normalization with the following:

Both run numbers: 59039
Sample: La11B6_NIST_660c_001
Grouping File: Column

Complete and save the results.

Run reduction with the 59039 run number and verify the workspaces with "output" in the name are about 3 GB or more.

Now repeat everything but on the reduce-data-size branch. You should not have to repeat the Normalization step. When inspecting the workspace/file size, you should see that they no longer take up 3 GB of space. The workspaces should show something like 50 MB and the tar file will be about 300 MB.

CIS testing

Same as the dev testing, just make sure all workspaces are still as expected.

Link to EWM item

EWM#6101

Added flag to not preserve events for DiffCal and Reduction workflows
@codecov
Copy link

codecov bot commented Jul 30, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Please upload report for BASE (staging@3e858ea). Learn more about missing BASE report.

Additional details and impacted files
@@            Coverage Diff             @@
##             staging     #425   +/-   ##
==========================================
  Coverage           ?   95.40%           
==========================================
  Files              ?       60           
  Lines              ?     3830           
  Branches           ?        0           
==========================================
  Hits               ?     3654           
  Misses             ?      176           
  Partials           ?        0           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@darshdinger darshdinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing on staging:
image

Testing on this branch:
image

Works as stated.

@dlcaballero16 dlcaballero16 merged commit 4d5c5e6 into staging Jul 30, 2024
@dlcaballero16 dlcaballero16 deleted the reduce-data-size branch July 30, 2024 18:27
walshmm pushed a commit that referenced this pull request Jul 30, 2024
Added flag to not preserve events for DiffCal and Reduction workflows
dlcaballero16 added a commit that referenced this pull request Jul 31, 2024
Added flag to not preserve events for DiffCal and Reduction workflows
@dlcaballero16 dlcaballero16 mentioned this pull request Jul 31, 2024
dlcaballero16 added a commit that referenced this pull request Aug 1, 2024
Added flag to not preserve events for DiffCal and Reduction workflows
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants