Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bpo-39584: raise ValueError when creating shared memory of size greater than 1TB #21877

Closed
wants to merge 1 commit into from
Closed

bpo-39584: raise ValueError when creating shared memory of size greater than 1TB #21877

wants to merge 1 commit into from

Conversation

vinay0410
Copy link
Contributor

@vinay0410 vinay0410 commented Aug 14, 2020 •
edited by bedevere-bot

@vinay0410 vinay0410 requested a review from tiran as a code owner August 14, 2020 09:11
@vinay0410 vinay0410 changed the title bpo-39584: raise ValueError when creating shared memory of greater than 1TB bpo-39584: raise ValueError when creating shared memory of size greater than 1TB Aug 15, 2020
@vinay0410
Copy link
Contributor Author

Hi @ronaldoussoren , I have made changes to python mulitprocessing.shared_memory.SharedMemory. Can you please review this.

@vinay0410 vinay0410 closed this Aug 28, 2020
@vinay0410
Copy link
Contributor Author

reopened PR to re-trigger CI

@vinay0410 vinay0410 reopened this Aug 28, 2020
ZackerySpytz
ZackerySpytz reviewed Aug 31, 2020
View reviewed changes
Misc/NEWS.d/next/Library/2020-08-14-14-31-04.bpo-39584.P1Rs2u.rst
@@ -0,0 +1,2 @@
Prevent creating :class:`shared_memory.SharedMemory` objects with
:code:`size` greater than 1TB.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is a valid use of the code role.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you have a look at this: #21556 (comment)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. Okay. But double backquotes are usually used in cases like this (as specified by the developer's guide https://devguide.python.org/documenting/#inline-markup).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess both of them work. See this https://devguide.python.org/documenting/#module-specific-markup .
I was going for this because it is tried and tested.
Although, if you still want me to change to double back ticks, I will make changes.

Lib/multiprocessing/shared_memory.py Outdated Show resolved Hide resolved
Lib/test/_test_multiprocessing.py
@@ -3880,6 +3880,10 @@ class OptionalAttachSharedMemory(shared_memory.SharedMemory):

sms.close()

# Test creating a shared memory segment with size greater than shared_memory._MAX_SHM_SIZE_LIMIT
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there's a need for this comment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added this comment because similar tests around this test use such comments.

Lib/test/_test_multiprocessing.py Outdated
@@ -3880,6 +3880,10 @@ class OptionalAttachSharedMemory(shared_memory.SharedMemory):

sms.close()

# Test creating a shared memory segment with size greater than shared_memory._MAX_SHM_SIZE_LIMIT
with self.assertRaises(ValueError):
sms_invalid = shared_memory.SharedMemory(create=True, size=shared_memory._MAX_SHM_SIZE_LIMIT + 1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no need for a variable here.

Copy link
Contributor Author

@vinay0410 vinay0410 Sep 3, 2020 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Totally agree, I will remove that.
But, if you see there are similar tests around this test assigning an unnecessary variable, should I remove those too ? Or should raise another pull request removing those.

Doc/library/multiprocessing.shared_memory.rst Outdated Show resolved Hide resolved
@vinay0410 vinay0410 force-pushed the fix-issue-39584 branch 2 times, most recently from 6eaa6da to 691f27c Compare September 3, 2020 10:40
ronaldoussoren
ronaldoussoren requested changes Oct 20, 2020
View reviewed changes
Copy link
Contributor

@ronaldoussoren ronaldoussoren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR changes the limit for all systems, not just on macOS. I'd prefer to have the limit only on macOS, the behaviour of other unix-y systems (and in particular Linux) is different and likely don't need this artificial limit.

Lib/multiprocessing/shared_memory.py
if not isinstance(item, (str, bytes))
else self._types_mapping[type(item)] % (
self._alignment * (len(item) // self._alignment + 1),
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't mix actual changes with formatting changes.

Lib/multiprocessing/shared_memory.py
@@ -422,7 +431,8 @@ def _set_packing_format_and_transform(self, position, fmt_as_str, value):
def __getitem__(self, position):
position = position if position >= 0 else position + self._list_len
try:
offset = self._offset_data_start + self._allocated_offsets[position]
offset = self._offset_data_start + \
self._allocated_offsets[position]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise

Lib/multiprocessing/shared_memory.py
@@ -449,7 +459,8 @@ def __setitem__(self, position, value):
new_format = self._types_mapping[type(value)]
encoded_value = value
else:
allocated_length = self._allocated_offsets[position + 1] - item_offset
allocated_length = self._allocated_offsets[position +
1] - item_offset
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise

@bedevere-bot
Copy link

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@ronaldoussoren
Copy link
Contributor

Also: I don't feel qualified to approve changes to multiprocessing. This is a complex library that I don't use myself.

tiran
tiran requested changes Nov 17, 2020
View reviewed changes
Copy link
Member

@tiran tiran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please read my most recent comment on bpo. 1 TB is arbitrary and far too low.

@vinay0410 vinay0410 mannequin mentioned this pull request Apr 10, 2022
Open
@iritkatriel
Copy link
Member

I'm closing this PR since the solution here has been rejected on the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ZackerySpytz ZackerySpytz ZackerySpytz left review comments

@tiran tiran tiran requested changes

@ronaldoussoren ronaldoussoren ronaldoussoren requested changes

Assignees
No one assigned
Labels
awaiting changes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
8 participants
@vinay0410 @bedevere-bot @ronaldoussoren @iritkatriel @tiran @ZackerySpytz @the-knights-who-say-ni @ezio-melotti