Add support for allocators in `LinkedList` #103093

rytheo · 2022-10-15T20:10:46Z

Allows LinkedList to use a custom allocator

rust-highfive · 2022-10-15T20:10:49Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @Mark-Simulacrum (or someone else) soon.

Please see the contribution instructions for more information.

rytheo · 2022-10-15T20:16:56Z

I'm having trouble specializing on SpecExtend. The problematic code is below:

impl<I: IntoIterator, A: Allocator + Clone> SpecExtend<I> for LinkedList<I::Item, A> {
    default fn spec_extend(&mut self, iter: I) {
        iter.into_iter().for_each(move |elt| self.push_back(elt));
    }
}

impl<T, A: Allocator + Clone> SpecExtend<LinkedList<T, A>> for LinkedList<T, A> {
    fn spec_extend(&mut self, ref mut other: LinkedList<T, A>) {
        self.append(other);
    }
}

The compiler doesn't like that A is repeated in the specializing impl. I could use advice on whether this is currently fixable.

Nilstrieb · 2022-10-16T06:53:21Z

library/alloc/src/collections/linked_list.rs

        }
    }
 }

 #[stable(feature = "rust1", since = "1.0.0")]
-impl<T> Default for LinkedList<T> {
+impl<T, A: Allocator + Clone + Default> Default for LinkedList<T, A> {


We usually just implement default for Global instead of all allocators, because that would cause inference failures in many cases.

Concretely, every program using LinkedList::default() that doesn't explicitly assign the list to LinkedList<_> (for example by passing it to a function or storing it in a field) will not infer an allocator, as type inference does not take into account default type parameters.

Mark-Simulacrum · 2022-10-16T16:28:34Z

I don't know enough about specialization to say whether there's a better option there. It looks like similar impls for Vec avoid a direct Vec<T, A> impl, but I would need to do more digging to say more than that.

I would recommend starting without this specialization, I think.

@rustbot author

rustbot · 2022-10-18T02:29:57Z

Hey! It looks like you've submitted a new PR for the library teams!

If this PR contains changes to any rust-lang/rust public library APIs then please comment with @rustbot label +T-libs-api -T-libs to tag it appropriately. If this PR contains changes to any unstable APIs please edit the PR description to add a link to the relevant API Change Proposal or create one if you haven't already. If you're unsure where your change falls no worries, just leave it as is and the reviewer will take a look and make a decision to forward on if necessary.

Examples of T-libs-api changes:

Stabilizing library features
Introducing insta-stable changes such as new implementations of existing stable traits on existing stable types
Introducing new or changing existing unstable library APIs (excluding permanently unstable features / features without a tracking issue)
Changing public documentation in ways that create new stability guarantees
Changing observable runtime behavior of library APIs

rytheo · 2022-10-18T02:31:59Z

@rustbot label -S-waiting-on-author +S-waiting-on-review

Mark-Simulacrum · 2022-11-06T23:22:35Z

library/alloc/src/collections/linked_list.rs

        })
    }

    /// Adds the given node to the back of the list.
    #[inline]
-    fn push_back_node(&mut self, mut node: Box<Node<T>>) {
+    fn push_back_node(&mut self, node: Node<T>) {
+        let mut node = Box::new_in(node, &self.alloc);


I'm worried that changes like this may cause extra stack space etc to be used for moving the Node value around, since there's more places where we're passing it prior to sticking it in the Box. I'm not sure it's worth blocking over, but wanted to raise a note of caution.

I've thought of some ways to avoid moving the node onto the stack. Passing &mut self, mut node: Box<Node<T>, &A> doesn't work because we can't mutably borrow the whole list while the box is borrowing the list's allocator. Unless we want to add a Clone bound on the allocator, we'll have to change one of the arguments.

The method doesn't mutate the allocator, so we could change the signature to take head, tail, len (or some struct containing them) instead of the whole list struct.

Alternatively, we could pass in a NonNull<Node<T>> instead of a Box.

@Mark-Simulacrum would the following signature be acceptable?
unsafe fn push_back_node(&mut self, node: Unique<Node<T>>)

I haven't found a way for this method to safely take a Box<Node<T>, &A>, since:

The box can't hold &self.alloc when the function already takes &mut self

The box might use a different instance of A with a different memory pool

It seems to me that the caller of this function can call Box::new_in, reaching into the LinkedList for the actual allocator, and then pass in node: Box<Node<T>, A> just fine, right? Otherwise none of this would actually work.

Or I guess NonNull<Node<T>> would have to be it. In any case, I'm inclined to not block on this particular function - we can figure that out at a later point. I think focusing on the other comments around changes in behavior is more important.

Unless we want to add a Clone bound on the allocator,

FWIW, BTreeMap has such a bound to avoid &A allocators -- those take up space in the Box even for the common case where A is a ZST. (I still think we should remove the blanked impl of &Allocator to avoid people accidentally blowing up their boxes to twice their size... though in this case it's only temporaries within a function so it's not so bad.)

Mark-Simulacrum · 2022-11-06T23:24:50Z

library/alloc/src/collections/linked_list.rs

@@ -639,7 +679,7 @@ impl<T> LinkedList<T> {
    #[inline]
    #[stable(feature = "rust1", since = "1.0.0")]
    pub fn clear(&mut self) {
-        *self = Self::new();
+        while self.pop_front_node().is_some() {}


I haven't looked at the Drop impl, but this seems less than optimal - previously we were presumably dropping the T in place rather than copying (moving it) out and then dropping it.

Note that this is really caused by the change to the pop function returning Node rather than the Box of a Node.

This also means that we're losing the panic "continue dropping" behavior of the drop impl.

Good point; I'll see if there's a way to avoid the excess copies.

Would it be reasonable to move the node drop code into clear and then have the drop impl call clear so they both get the panic guard?

Seems plausible.

Mark-Simulacrum · 2022-11-06T23:30:38Z

@rustbot author

rytheo · 2022-11-15T23:40:52Z

@Mark-Simulacrum, is it safe to assume two non-clone instances of an allocator type manage the same memory pool? If not, then we may not be able to keep O(1) time for append, since we'd have to loop and move all elements from one pool into another.

Additionally: If multiple allocators of a type can have different pools, we might need unsafe versions of the private push/pop boxed node methods since the caller would have to guarantee the box and list allocators use the same pool.

Mark-Simulacrum · 2022-11-30T22:59:50Z

I'm not sure what you mean by non-clone instances of an allocator type - but in general I think the assumption that the same allocator type implies a common memory pool shouldn't be guaranteed. Most of the per-datastructure custom allocator work is pretty early days though.

I don't think we would want to change append from O(1) to not O(1) in this PR; unfortunately, I'm not sure we can readily detect the situation where that optimization doesn't work with today's allocator API. It's probably a good idea to talk with folks working on it to see if maybe we need an fn is_same_as or something method...

cc @Amanieu -- I seem to recall you being involved with the allocator work

Amanieu · 2022-12-01T09:15:00Z

I think this makes a good case for adding trait Allocator: Eq so that you can check whether two allocators are compatible and can reuse each other's memory. I've opened a issue in wg-allocator for this: rust-lang/wg-allocators#109

Regarding the append function, I don't think we need to support an O(n) implementation: we can just panic if the 2 allocators are incompatible.

For now, you can leave append as only implemented for A = Global. The same applies to CursorMut::splice_*. Once Eq is added for allocators then these can be changed to be generic over A with an assertion that the allocators are the same.

bors · 2023-04-03T04:04:00Z

⌛ Testing commit c8d74c0feeb060dffcd70bce0c27e3255e07c527 with merge b6c88315c4c3e45ea9c10fef33ca21b1acdc2bce...

bors · 2023-04-03T04:53:55Z

💔 Test failed - checks-actions

Mark-Simulacrum · 2023-04-24T12:30:43Z

r=me with commits squashed (and please keep them squashed, no need for amendment commits in the future)

rytheo · 2023-04-25T04:10:24Z

@bors r=Mark-Simulacrum

bors · 2023-04-25T04:10:26Z

@rytheo: 🔑 Insufficient privileges: Not in reviewers

ChrisDenton · 2023-04-25T05:27:16Z

@bors r=Mark-Simulacrum

bors · 2023-04-25T05:27:17Z

📌 Commit 34136ab has been approved by Mark-Simulacrum

It is now in the queue for this repository.

bors · 2023-04-25T11:35:02Z

⌛ Testing commit 34136ab with merge 20d90b1...

bors · 2023-04-25T13:45:11Z

☀️ Test successful - checks-actions
Approved by: Mark-Simulacrum
Pushing 20d90b1 to master...

rust-timer · 2023-04-25T15:31:43Z

Finished benchmarking commit (20d90b1): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	1.9%	[1.9%, 1.9%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Cycles

This benchmark run did not return any relevant results for this metric.

RalfJung · 2023-07-19T18:23:16Z

library/alloc/src/collections/linked_list.rs

    #[inline]
-    fn push_front_node(&mut self, mut node: Box<Node<T>>) {
+    unsafe fn push_front_node(&mut self, node: Unique<Node<T>>) {


Ah, we were actually quite happy that Unique is basically unused in the standard library. It'd be better to avoid introducing new uses of this type until its status is clarified. Currently it makes absolutely no difference to NonNull. There are some vague plans to maybe attach alias information to that type, but currently this does not happen. (Cc rust-lang/unsafe-code-guidelines#384)

Why was the type chosen here?

Unique seemed like a better fit since this method takes ownership of the node. Changing the signature to use NonNull should be fine; this doc comment should probably also mention that the pointer shouldn’t be used again.

Unique is Copy, it can't really indicate ownership.

…=cuviper Avoid using `ptr::Unique` in `LinkedList` code Addresses a [comment](rust-lang#103093 (comment)) by `@RalfJung` about avoiding use of `core::ptr::Unique` in the standard library.

Avoid using `ptr::Unique` in `LinkedList` code Addresses a [comment](rust-lang/rust#103093 (comment)) by `@RalfJung` about avoiding use of `core::ptr::Unique` in the standard library.

rust-highfive assigned Mark-Simulacrum Oct 15, 2022

rustbot added the T-libs Relevant to the library team, which will review and decide on the PR/issue. label Oct 15, 2022

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 15, 2022

This comment has been minimized.

Sign in to view

Nilstrieb reviewed Oct 16, 2022

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 16, 2022

rytheo force-pushed the linked-list-alloc-api branch from 0ef1a30 to b0c0199 Compare October 18, 2022 01:46

This comment has been minimized.

Sign in to view

rytheo marked this pull request as ready for review October 18, 2022 02:29

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 18, 2022

rytheo mentioned this pull request Oct 18, 2022

Tracking Issue for structs which needs an allocator rust-lang/wg-allocators#7

Open

15 tasks

rytheo force-pushed the linked-list-alloc-api branch from 767c640 to ed58eb5 Compare October 18, 2022 03:45

rytheo requested a review from Nilstrieb October 19, 2022 01:19

Mark-Simulacrum reviewed Nov 6, 2022

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 6, 2022

Amanieu mentioned this pull request Dec 1, 2022

Allocator equality rust-lang/wg-allocators#109

Open

Mark-Simulacrum added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Mar 4, 2023

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 2, 2023

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Apr 3, 2023

This comment has been minimized.

Sign in to view

Add support for allocators in LinkedList

34136ab

rytheo force-pushed the linked-list-alloc-api branch from c0e7995 to 34136ab Compare April 25, 2023 02:33

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 25, 2023

bors added the merged-by-bors This PR was explicitly merged by bors. label Apr 25, 2023

bors merged commit 20d90b1 into rust-lang:master Apr 25, 2023

rustbot added this to the 1.71.0 milestone Apr 25, 2023

rytheo deleted the linked-list-alloc-api branch April 25, 2023 23:03

RalfJung reviewed Jul 19, 2023

View reviewed changes

rytheo mentioned this pull request Jul 30, 2023

Avoid using ptr::Unique in LinkedList code #114257

Merged

Add support for allocators in LinkedList #103093

Add support for allocators in LinkedList #103093

Conversation

rytheo commented Oct 15, 2022

rust-highfive commented Oct 15, 2022

This comment has been minimized.

rytheo commented Oct 15, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mark-Simulacrum commented Oct 16, 2022 • edited Loading

This comment has been minimized.

rustbot commented Oct 18, 2022

rytheo commented Oct 18, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mark-Simulacrum commented Nov 6, 2022

rytheo commented Nov 15, 2022

Mark-Simulacrum commented Nov 30, 2022

Amanieu commented Dec 1, 2022

bors commented Apr 3, 2023

bors commented Apr 3, 2023

This comment has been minimized.

Mark-Simulacrum commented Apr 24, 2023

rytheo commented Apr 25, 2023

bors commented Apr 25, 2023

ChrisDenton commented Apr 25, 2023

bors commented Apr 25, 2023

bors commented Apr 25, 2023

bors commented Apr 25, 2023

rust-timer commented Apr 25, 2023

Overall result: no relevant changes - no action needed

Choose a reason for hiding this comment

rytheo Jul 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Add support for allocators in `LinkedList` #103093

Add support for allocators in `LinkedList` #103093

rytheo commented Oct 15, 2022 •

edited

Loading

Mark-Simulacrum commented Oct 16, 2022 •

edited

Loading

rytheo Jul 20, 2023 •

edited

Loading