random.shuffle loses most of the elements

[rowanbradley]

BPO	43618
Nosy	@tim-one, @rhettinger, @scoder, @ericvsmith

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-03-25.18:45:21.607>
created_at = <Date 2021-03-24.18:30:46.123>
labels = ['type-bug', 'library', '3.9']
title = 'random.shuffle loses most of the elements'
updated_at = <Date 2021-03-25.21:22:24.250>
user = 'https://bugs.python.org/rowanbradley'

bugs.python.org fields:

activity = <Date 2021-03-25.21:22:24.250>
actor = 'rhettinger'
assignee = 'none'
closed = True
closed_date = <Date 2021-03-25.18:45:21.607>
closer = 'scoder'
components = ['Library (Lib)']
creation = <Date 2021-03-24.18:30:46.123>
creator = 'rowan.bradley'
dependencies = []
files = []
hgrepos = []
issue_num = 43618
keywords = []
message_count = 7.0
messages = ['389482', '389483', '389485', '389509', '389510', '389518', '389530']
nosy_count = 5.0
nosy_names = ['tim.peters', 'rhettinger', 'scoder', 'eric.smith', 'rowan.bradley']
pr_nums = []
priority = 'normal'
resolution = 'third party'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43618'
versions = ['Python 3.9']

[rowanbradley]

This issue is probably related to issue ??? but I have created it as a separate issue. When shuffle doesn't crash it sometimes (or maybe always - I haven't fully analysed this yet) looses most of the elements in the list that it is supposed to be shuffling. Here is an extract of the code that I'm using:

import io
from io import StringIO 
from lxml import etree 
import random  

filename_xml = 'MockExam5.xml'
with io.open(filename_xml, mode="r", encoding="utf-8") as xml_file:
    xml_to_check = xml_file.read()
doc = etree.parse(StringIO(xml_to_check))
exams = doc.getroot()
questions_element = exams.find("questions")
logmsg(L_TRACE, "There are now " + str(len(questions_element.findall("question"))) + " questions")
logmsg(L_TRACE, "Randomising order of questions in this exam")
random.shuffle(questions_element)
logmsg(L_TRACE, "Finished randomise")
logmsg(L_TRACE, "There are now " + str(len(questions_element.findall("question"))) + " questions")

And here is the log produced by this code:

21-03-24 18:10:11.989 line: 2057 file: D:\XPS_8700 Extended Files\Users\RowanB\Documents\My_Scripts NEW\mockexam\put_exam.py 2 There are now 79 questions
21-03-24 18:10:11.991 line: 2065 file: D:\XPS_8700 Extended Files\Users\RowanB\Documents\My_Scripts NEW\mockexam\put_exam.py 2 Randomising order of questions in this exam
21-03-24 18:10:11.992 line: 2067 file: D:\XPS_8700 Extended Files\Users\RowanB\Documents\My_Scripts NEW\mockexam\put_exam.py 2 Finished randomise
21-03-24 18:10:11.993 line: 2068 file: D:\XPS_8700 Extended Files\Users\RowanB\Documents\My_Scripts NEW\mockexam\put_exam.py 2 There are now 6 questions

How come the shuffle starts off with 79 elements, and finishes with 6?

Thanks - Rowan

[ericvsmith]

Same advice as bpo-43616: please provide an example we can run.

This most likely a problem with how you're using lxml, and not a bug in python. But since we can't test it, we can't know for sure.

[tim-one]

Are you sure it's "a list"? At least print out type(questions_element). random.shuffle() doesn't contain any code capable of changing a list's length. It only does indexed accessing of the list:

...
for i in reversed(range(1, len(x))):
# pick an element in x[:i+1] with which to exchange x[i]
j = randbelow(i + 1)
x[i], x[j] = x[j], x[i]

That's about all there is to it. Note that, for this purpose, it doesn't matter want randbelow() does, because that function never even sees x.

[rhettinger]

The standard library isn't at fault here. Please file this an an LXML bug.

Reproducer:

    from lxml.etree import Element

    root = Element('outer')
    root.append(Element('zero'))
    root.append(Element('one'))
    root.append(Element('two'))
    print([e.tag for e in root])
    root[1], root[0] = root[0], root[1]
    print([e.tag for e in root])

This outputs:

['zero', 'one', 'two']
['one', 'two']

Replacing the import with:

   from xml.etree.ElementTree import Element

Gives the expected result:

['zero', 'one', 'two']
['one', 'zero', 'two']

[rhettinger]

Interestingly, this isn't an LXML bug. It is a documented difference from how the standard library works:

https://lxml.de/tutorial.html#elements-are-lists

So, if you want use random.shuffle(), you need the standard library ElementTree instead of lxml.

This:

    from lxml.etree import Element

    root = Element('outer')
    root.append(Element('zero'))
    root.append(Element('one'))
    root.append(Element('two'))
    root[0] = root[1]
    print([e.tag for e in root])

Produces:

['one', 'two']

[scoder]

Yes, this is neither a bug in CPython (or its stdlib) nor in lxml. It's how things work. Don't use these two together.

[rhettinger]

As an immediate fix to your problem, replace this line:

    random.shuffle(questions_element)

with:

    questions = list(questions_element)
    random.shuffle(questions)
    questions_element[:] = questions