Skip to content

Deep copy from Sidre to Conduit#1510

Merged
gunney1 merged 83 commits intodevelopfrom
feature/gunney/copy-sidre-to-conduit
Sep 21, 2025
Merged

Deep copy from Sidre to Conduit#1510
gunney1 merged 83 commits intodevelopfrom
feature/gunney/copy-sidre-to-conduit

Conversation

@gunney1
Copy link
Contributor

@gunney1 gunney1 commented Mar 3, 2025

Summary

  • This PR is a feature
  • It does the following:
    • Implement Group::deepCopyToConduit and View::deepCopyToConduit
    • Secondarily, but critically, provide a mechanism for allocating Views with different allocators based on the View type (string, scalar or array). The mechanism generalizes scalars so they can be array-like but treated as scalars for the purpose of data allocation and copying. In use, this can transparently place big computational data on device and metadata on host.
    • Support copying from one allocator id to another. The allocator ids are set before the copy, and the copy assures destination memory uses the right id.
    • Support Conduit memory allocation using any Axom or Umpire allocator id. This functionality partially overlaps the one recently added to the bump component. We will combine the two after this merge.
    • Fix bug that always allocates string and scalar View data on host. They now allocate in the default allocator of the Group that owns the View. (We have discussed this in the project meetings, reaching the concensus was that all data should use the default allocator unless individually overridden, despite the fact that many parts of a hierarchy would only be used on the host.) Preserves the unintended but useful behavior of scalars and strings always being allocated on the host. Scalars and strings now have their own allocator, with the option of being placed on device if the user wants.
    • Manually tell a bunch of Views to allocate data on the host, where they are used and expected to be.
    • Add tests for the new methods.

The generalized scalar type is called a "tuple", which is treated like a scalar but may have more than one element. This change:

  • is backward compatible
  • preserves the current unintended but useful sidre behavior placing strings and scalars on hosts
  • continues to support the previously intended behavior of putting strings, scalars and arrays in the same memory space, if that's what the user wants
  • adds a small set of critical methods: View::setTuple, View::isTuple, Group::setDefaultArrayAllocator and Group::setDefaultTupleAllocator
  • changes Group::setDefaultAllocator and Group::getDefaultAllocator to call the array versions of those methods

The use of tuples greatly simplifies working with sidre hierarchies. Using the same allocator for all data and manually overriding individual leaves, was not workable.

The goal of this change is toward supporting transparent switching between sidre and conduit. With the addition of Group::deepCopyToConduit, the state of conduit-sidre hierarchy copies is:
Screenshot_2025-04-22_06-02-48
* I labeled importConduitTreeExternal as the shallow conduit-to-sidre copy, but it's not exactly analogous to createNativeLayout (the shallow sidre-to-conduit copy).

@gunney1 gunney1 added the Sidre Issues related to Axom's 'sidre' component label Mar 3, 2025
@gunney1 gunney1 self-assigned this Mar 3, 2025
gunney1 added 20 commits March 6, 2025 09:17
The actual fix is on another branch, but I can't wait for it to
be merged.
It's failing.  Maybe because non-Umpire allocators are not
working right or Group isn't using it right.
Also disable test copying in/out of Umpire.
This check needs to wait for proper support for
that kind of copy, which is being performed on
another branch.
change transfer_allocator method name to transferTo.
Also use call-back to return View-specific allocator.
@gunney1 gunney1 force-pushed the feature/gunney/copy-sidre-to-conduit branch from 6764c24 to 41c15d7 Compare March 28, 2025 18:08
@gunney1 gunney1 force-pushed the feature/gunney/copy-sidre-to-conduit branch from 582bbad to 46bb8a7 Compare April 1, 2025 01:06
@gunney1 gunney1 requested a review from kennyweiss August 7, 2025 20:54
@rhornung67
Copy link
Member

Please review this PR as it must be merged soon!

@gunney1 gunney1 added this to the 2025 Fall Release milestone Sep 10, 2025
@gunney1
Copy link
Contributor Author

gunney1 commented Sep 11, 2025

This PR must be completed before #1654. Both are required for the 2025 Fall release.

@gunney1 gunney1 closed this Sep 11, 2025
@gunney1 gunney1 reopened this Sep 11, 2025
Copy link
Member

@kennyweiss kennyweiss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @gunney1 -- please clean up the primal/klee stuff that leaked in from your other PR

Copy link
Contributor

@Arlie-Capps Arlie-Capps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gunney1 , thank you for your hard work on this.

Copy link
Member

@kennyweiss kennyweiss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @gunney1

struct ConduitMemory
{
//!@brief Return the Axom allocator id.
int axomId() const { return m_axomId; }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consistency: Do/Should we use Id or ID or id throughout Axom?

It doesn't look like we've been especially consistent, but we might want to consider this in the future.
See: https://sourcegraph.com/search?q=repo:%5Egithub%5C.com/llnl/axom+Id&patternType=keyword&sm=0


// Note: most of these calls that set the view class members are
// unnecessary if the view already holds a tuple. May be
// a future optimization opportunity to split the
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please finish this sentence.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually don't know how to finish it. It appears multiple times in the file. Aaron wrote the comment and I used his existing methods as a template. I'll remove the fragments.

@gunney1 gunney1 merged commit a60520d into develop Sep 21, 2025
15 checks passed
@gunney1 gunney1 deleted the feature/gunney/copy-sidre-to-conduit branch September 22, 2025 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

GPU Issues related to GPU development Reviewed Sidre Issues related to Axom's 'sidre' component

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants