Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Manage dataset configuration #53

Merged
merged 31 commits into from
Jul 26, 2023
Merged

Manage dataset configuration #53

merged 31 commits into from
Jul 26, 2023

Conversation

rsreds
Copy link
Collaborator

@rsreds rsreds commented Feb 7, 2023

Prototype implementation of a system to manage datasets.
Uses json file to define datasets to be used for data creation in Storage Services, then accessed by jobs.

#17

@rsreds rsreds changed the title Manage dataset configuration Draft: Manage dataset configuration Feb 7, 2023
@rsreds
Copy link
Collaborator Author

rsreds commented Mar 1, 2023 •
edited
Loading

  • Manage workload with no infile
  • Put assignfiles as a method of workload
  • Handle multiple dataset for workload (concatenate files vector)

@rsreds rsreds marked this pull request as ready for review April 11, 2023 14:56
@rsreds
Copy link
Collaborator Author

rsreds commented Apr 14, 2023 •
edited
Loading

  • Remove CLI options
  • Merge workload constructors
  • Remove files epr job argument
  • Fix .sh script and add config files for those

@rsreds rsreds marked this pull request as draft April 14, 2023 09:12
@rsreds rsreds marked this pull request as ready for review April 17, 2023 09:39
HerrHorizontal
HerrHorizontal previously requested changes Apr 19, 2023
View reviewed changes
Copy link
Collaborator

@HerrHorizontal HerrHorizontal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just have some minor comments and requests (see comments).

src/Dataset.cpp Show resolved Hide resolved
src/Workload.cpp Show resolved Hide resolved
src/Workload.cpp Show resolved Hide resolved
src/SimpleSimulator.cpp Show resolved Hide resolved
src/SimpleSimulator.cpp Show resolved Hide resolved
src/SimpleSimulator.cpp Show resolved Hide resolved
@HerrHorizontal HerrHorizontal changed the title Draft: Manage dataset configuration Manage dataset configuration Apr 19, 2023
rsreds added 16 commits July 26, 2023 11:01
@rsreds
Add location to streamed computation log
6e8af5c
@rsreds
Change map to vector
7e9842a
@rsreds
Add initial implementation
3b65be7 
Add dataset_configuration option for main
Create Dataset class for dataset specs

Doesn't do anything for now
@rsreds
Add file generation in Dataset constructor
ce7fa9d
@rsreds
First implementation of dataset file store
03242a5 
Not working.
Missing a way to specify the files from the workload.
@rsreds
Add files assignment to jobs
817dc37
@rsreds
Add initial implementation
839705c 
Add dataset_configuration option for main
Create Dataset class for dataset specs

Doesn't do anything for now
@rsreds
Add file generation in Dataset constructor
7c7d0cf
@rsreds
First implementation of dataset file store
4b812af 
Not working.
Missing a way to specify the files from the workload.
@rsreds
Add files assignment to jobs
5b13de4
@rsreds
Add support for new RNG in dataset creatin
3585643
@rsreds
Move assignFiles to workload class
8648963
@rsreds
Restore correct file instantiation
fdfcc9a
@rsreds
Add support for multiple dataset per workload
b4bb47b
@rsreds
Improve handling of json type
877b9e4 
For handling strings or array of strings for datasets and workload
configuration
@rsreds
Remove unnecessary copy operations
382d0a1
rsreds added a commit to rsreds/DCSim that referenced this pull request Jul 26, 2023
@rsreds
Fix after review
e01786f 
HEPCompSim#53 (review)
@rsreds
Copy link
Collaborator Author

rsreds commented Jul 26, 2023 •
edited
Loading

Should merge after #61

HerrHorizontal
HerrHorizontal approved these changes Jul 26, 2023
View reviewed changes
Copy link
Collaborator

@HerrHorizontal HerrHorizontal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finally, it is done! Congratulations to us all!

@HerrHorizontal HerrHorizontal merged commit 04f2035 into HEPCompSim:main Jul 26, 2023
@HerrHorizontal HerrHorizontal mentioned this pull request Jul 26, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HerrHorizontal HerrHorizontal HerrHorizontal approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@rsreds @HerrHorizontal