rocAL Deserialize PR 2 : Initialize Arguments for Nodes #432

fiona-gladwin · 2025-12-15T18:57:17Z

Motivation

Introduce support to extract argument extraction and initialize the Node during deserialization.

Technical Details

Adds a get() method to the Argument class for type-safe value retrieval and implements
Introduce initialize_args() methods in node classes to support deserialization-based initialization.
Add support for initialize_args() in ImageLoaderNode, ImageSingleShardNode and BrightnessNode

Test Plan

No new tests added. Must pass existing tests

NOTE : To be merged after PR #430

Remove the use of pipeline operator

…change

…ocAL into fg/ser_argument

Co-authored-by: spolifroni-amd <Sandra.Polifroni@amd.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

LakshmiKumar23 · 2026-01-09T21:21:27Z

@fiona-gladwin Can you resolve merge conflicts? Thanks!

codecov · 2026-01-12T19:59:19Z

Codecov Report

❌ Patch coverage is 2.24719% with 87 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rocAL/include/pipeline/argument.h	0.00%	49 Missing ⚠️
rocAL/source/loaders/image/node_image_loader.cpp	7.14%	13 Missing ⚠️
...e/loaders/image/node_image_loader_single_shard.cpp	7.69%	12 Missing ⚠️
rocAL/include/pipeline/node.h	0.00%	8 Missing ⚠️
...mentations/color_augmentations/node_brightness.cpp	0.00%	5 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #432      +/-   ##
===========================================
- Coverage    79.97%   79.66%   -0.31%     
===========================================
  Files          294      294              
  Lines        22560    22647      +87     
===========================================
  Hits         18041    18041              
- Misses        4519     4606      +87

Files with missing lines	Coverage Δ
...ugmentations/color_augmentations/node_brightness.h	`100.00% <ø> (ø)`
rocAL/include/loaders/image/node_image_loader.h	`33.33% <ø> (ø)`
...ude/loaders/image/node_image_loader_single_shard.h	`33.33% <ø> (ø)`
...mentations/color_augmentations/node_brightness.cpp	`82.50% <0.00%> (-11.79%)`	⬇️
rocAL/include/pipeline/node.h	`60.00% <0.00%> (-28.24%)`	⬇️
...e/loaders/image/node_image_loader_single_shard.cpp	`74.19% <7.69%> (-17.81%)`	⬇️
rocAL/source/loaders/image/node_image_loader.cpp	`73.77% <7.14%> (-19.98%)`	⬇️
rocAL/include/pipeline/argument.h	`48.61% <0.00%> (-25.07%)`	⬇️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

LakshmiKumar23 · 2026-01-12T21:22:15Z

CI passes: https://math-ci.amd.com/blue/organizations/jenkins/mainline%2Fprecheckin%2FrocAL/detail/PR-432/1/pipeline Can be merged

rrawther

Please address the review comments

rrawther · 2026-01-13T21:24:08Z

rocAL/source/loaders/image/node_image_loader.cpp

    reader_cfg.set_sharding_info(sharding_info);

-    std::array<std::string, 23> arg_names = {
+    std::array<std::string, INIT_ARGS_COUNT> arg_names = {


Please define as const std::string arg_array[] = {...}
and use size(arguments) to get size if needed. This is not recommended

Since we are using std::array we have used the INIT_ARGS_COUNT to create the static array

rrawther · 2026-01-13T21:24:45Z

rocAL/source/loaders/image/node_image_loader.cpp


 #include "pipeline/exception.h"

+#define INIT_ARGS_COUNT 23  // Modify in accordance with number of args in init


remove hardcoded parameter. see comment below

@rrawther The INIT_ARGS_COUNT is used to ensure that the number of args serialized is retrieved correctly during deserialize.

I suggest adding a vector "std::vectorstd::string arguments" as part of the class and just copy the passed vector into the class member.
After that, you can just check the size of vector to get the size instead of hardcoded value

rrawther · 2026-01-13T21:26:19Z

rocAL/source/loaders/image/node_image_loader.cpp

 }

+void ImageLoaderNode::initialize_args(std::vector<Argument> &arguments, std::shared_ptr<MetaDataReader> meta_data_reader) {
+    if (arguments.size() != INIT_ARGS_COUNT)


use arg_array.size() instead

rrawther · 2026-01-13T21:26:38Z

rocAL/source/loaders/image/node_image_loader_single_shard.cpp

    reader_cfg.set_sharding_info(sharding_info);

-    std::array<std::string, 23> arg_names = {
+    std::array<std::string, INIT_ARGS_COUNT> arg_names = {


please see the comment above

@rrawther The INIT_ARGS_COUNT is used to ensure that the number of args serialized is retrieved correctly during deserialize.

As in the initialize_args we need to check the required number of arguments are present to extract each of them and pass to the init function, If not it might lead to segmentation fault, hence we have hardcoded the value

rrawther · 2026-01-21T18:57:55Z

rocAL/include/loaders/image/node_image_loader.h

              DecoderType decoder_type, bool shuffle, bool loop, size_t load_batch_count, RocalMemType mem_type, std::shared_ptr<MetaDataReader> meta_data_reader, bool decoder_keep_orig = false, const ShardingInfo& sharding_info = ShardingInfo(), 
              const char *prefix = "", unsigned sequence_length = 0, unsigned step = 0, unsigned stride = 0, ExternalSourceFileMode external_file_mode = ExternalSourceFileMode::NONE, const std::string &index_path = "");

+    void initialize_args(std::vector<Argument> &arguments, std::shared_ptr<MetaDataReader> meta_data_reader) override;


please pass as const if the std::vector is read-only

rrawther

Added review comments

rrawther · 2026-01-21T19:05:15Z

rocAL/source/loaders/image/node_image_loader.cpp


 #include "pipeline/exception.h"

+#define INIT_ARGS_COUNT 23  // Modify in accordance with number of args in init


I suggest adding a vector "std::vectorstd::string arguments" as part of the class and just copy the passed vector into the class member.
After that, you can just check the size of vector to get the size instead of hardcoded value

rrawther · 2026-01-21T19:12:50Z

rocAL/source/loaders/image/node_image_loader.cpp

+    ShardingInfo sharding_info(arguments[13].get<RocalBatchPolicy>(), arguments[14].get<bool>(), arguments[15].get<bool>(), arguments[16].get<int32_t>());
+    std::string file_prefix = arguments[17].get<std::string>();
+
+    this->init(arguments[0].get<unsigned>(), arguments[1].get<unsigned>(), arguments[2].get<std::string>(),


to get rid of hardcoded arguments in Init(), I suggest adding a new Init() function which takes the passed vector of arguments directly. Also you need to have some way of checking the values against expected. If so it will fail the Init function

@rrawther As suggested I have removed the hardcoded value and introduced a map instead where the arguments are obtained from the map using the name of the argument (i.e string) Please let me know if this works

Store the args in a map in the Node Modify initializers to accept and obtain values from the ArgumentSet using the string

fiona-gladwin and others added 30 commits October 7, 2025 05:03

Merge branch 'fg/enum_change' into fg/enum_register

efe2b71

Merge branch 'fg/ser1_pipe_op' into fg/ser2_pipe_serialize

e736bfe

Add all serialization code to pipeline serializer

bef321b

Remove the use of pipeline operator

Merge branch 'fg/ser2_pipe_serialize' into fg/ser3_params_serialize

a82fb7f

Merge branch 'fg/ser3_params_serialize' into fg/serialize_test

5842b84

Revert fix

bbc96af

Update CHANGELOG.md

c4101fc

Update CHANGELOG.md

b668043

Update CHANGELOG.md

877bee0

Add comments and doc description

43636be

Support to serialize to file

64697e6

Update CMakeLists version for rocAL

ffdee4b

Merge branch 'fg/enum_change' into fg/enum_register

f9985cb

Merge branch 'develop' into fg/enum_register

01cc2d2

Merge branch 'develop' of https://github.com/ROCm/rocAL into fg/enum_…

7619bdd

…change

Merge branch 'fg/enum_change' into fg/enum_register

1d71cf9

Merge branch 'develop' into fg/enum_register

628ea36

Merge branch 'develop' into fg/enum_register

e620f0a

Merge branch 'develop' into fg/enum_register

5052fd6

Merge branch 'fg/enum_register' of https://github.com/fiona-gladwin/r…

aa91eed

…ocAL into fg/ser_argument

Update CHANGELOG

df3f598

Merge branch 'fg/ser_argument' into fg/ser1_pipe_op

1e88632

Update CHANGELOG.md

3077acb

Minor change

d2ccaab

Merge branch 'fg/ser_argument' into fg/ser1_pipe_op

6a7c102

Minor changes

59f7c0e

Update CHANGELOG.md

37fe4be

Co-authored-by: spolifroni-amd <Sandra.Polifroni@amd.com>

Merge branch 'fg/ser_argument' into fg/ser1_pipe_op

2be0388

Add const qualifier

29359b7

Merge branch 'fg/ser1_pipe_op' into fg/ser2_pipe_serialize

151455a

fiona-gladwin and others added 4 commits January 6, 2026 05:46

Minor change

74a346a

Apply suggestions from code review

6e0a28c

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Merge branch 'fg/serialize_python' into fg/deser_pr1

5a4a13b

Merge branch 'fg/deser_pr1' into fg/deser_pr2

464e5f7

fiona-gladwin dismissed spolifroni-amd’s stale review via 464e5f7 January 6, 2026 12:05

fiona-gladwin and others added 7 commits January 7, 2026 18:21

Merge branch 'develop' into fg/deser_pr1

890f8e6

Merge branch 'develop' into fg/deser_pr1

7e13d3c

Update CHANGELOG

ae296a3

Merge branch 'fg/deser_pr1' into fg/deser_pr2

88b25d6

Update CHANGELOG.md

18d4aba

Address review comments

363881a

Merge branch 'develop' into fg/deser_pr2

fe59174

fiona-gladwin and others added 3 commits January 12, 2026 14:53

Merge branch 'develop' into fg/deser_pr2

dc46b70

Minor change

169c1fe

Minor change

999c83e

LakshmiKumar23 previously approved these changes Jan 12, 2026

View reviewed changes

rrawther requested changes Jan 13, 2026

View reviewed changes

LakshmiKumar23 and others added 2 commits January 13, 2026 13:30

Merge branch 'develop' into fg/deser_pr2

cf3ac34

Merge branch 'develop' into fg/deser_pr2

008b37a

rrawther approved these changes Jan 21, 2026

View reviewed changes

Merge branch 'develop' into fg/deser_pr2

7c24e80

LakshmiKumar23 requested a review from rrawther January 21, 2026 18:50

rrawther reviewed Jan 21, 2026

View reviewed changes

rrawther requested changes Jan 21, 2026

View reviewed changes

Introduce ArgumentSet

3463633

Store the args in a map in the Node Modify initializers to accept and obtain values from the ArgumentSet using the string

fiona-gladwin dismissed LakshmiKumar23’s stale review via 3463633 January 23, 2026 16:40

Merge branch 'develop' into fg/deser_pr2

d3b7a36


		#include "pipeline/exception.h"

		#define INIT_ARGS_COUNT 23 // Modify in accordance with number of args in init

rocAL Deserialize PR 2 : Initialize Arguments for Nodes #432

Are you sure you want to change the base?

rocAL Deserialize PR 2 : Initialize Arguments for Nodes #432

Conversation

fiona-gladwin commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Test Plan

Uh oh!

LakshmiKumar23 commented Jan 9, 2026

Uh oh!

codecov bot commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

LakshmiKumar23 commented Jan 12, 2026

Uh oh!

rrawther left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fiona-gladwin Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rrawther left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

fiona-gladwin commented Dec 15, 2025 •

edited

Loading

codecov bot commented Jan 12, 2026 •

edited

Loading

fiona-gladwin Jan 16, 2026 •

edited

Loading