PageChooserBlock/ImageChooserBlock values go null when restoring from loaddata

[mx-moth]

If a database dump is taken using ./manage.py dumpdata > data.json, and then restored to an empty database (after taking care of #2217) using ./manage.py loaddata data.json, StreamField chooser blocks can loose their value.

This behaviour is dependent upon the order of the data from dumpdata, so is not 100% reproducible without some manual intervention. It does happen without manual intervention on occasion though.

My best guess is that StreamFields will validate their data for validity and integrity when they are unserialized. If the page instance a PageChooserBlock references, or the image an ImageChooserBlock references has not been loaded yet, the StreamField will replace the instance ID with null. When the loaddata import is finished, the database is not how it should be, and any ChooserBlocks that were loaded in the wrong order will be empty.

To reproduce:

• wagtail start testdata

• Add a StreamField with some chooser blocks to the HomePage:

  from wagtail.wagtailadmin.edit_handlers import StreamFieldPanel
  from wagtail.wagtailcore import blocks
  from wagtail.wagtailcore.fields import StreamField
  from wagtail.wagtailcore.models import Page
  from wagtail.wagtailimages.blocks import ImageChooserBlock
  
  
  class HomePage(Page):
  body = StreamField([
      ('page', blocks.PageChooserBlock()),
      ('image', ImageChooserBlock()),
  ])
  
  content_panels = Page.content_panels + [
      StreamFieldPanel('body'),
  ]

• ./manage.py makemigrations; ./manage.py migrate

• Edit the HomePage from the admin, choosing some pages and images in the StreamField.

• Dump, flush and load the data:

  $ ./manage.py dumpdata > data.json
  $ ./manage.py sqlflush | ./manage.py dbshell
  $ ./manage.py loaddata data.json

• Go back to the admin, or use ./manage.py dbshell, to observe that HomePage.body no longer correctly references the chosen pages and images. Instead, these chooser blocks are blank as if nothing has been chosen, with null in the fields instead of the page/image id.

As mentioned, this behaviour is dependant upon the order of the data dump, which is not consistent across runs. You can manually ensure the incorrect behaviour happens by editing data.json and moving the HomePage instances above the pages/images referenced in the StreamField. The rest of the data loading will work, including foreign keys to instances yet to be created.

[gasman]

I suspect this happens at the point where StreamBlock.get_prep_value is called, to write the StreamField value back to the database - this recursively calls get_prep_value on each child block, constructing the block value from the raw JSONish data in the process (if necessary), which here results in a useless round trip from JSONish data (= page/image ID) to native block value (= page/image instance) and back again.

A possible fix would be to make StreamBlock.get_prep_value aware of lazy StreamValues, to avoid the round trip:

• if StreamBlock.get_prep_value receives a StreamValue, call get_prep_value() or (get_raw_stream_data() or some other sensible name) on it
• StreamValue.get_prep_value() works as follows, for each block in the stream:
  • if this StreamValue is lazy and the item exists in self.stream_data but not self._bound_blocks (ref: Make StreamField mutable (issue #2876) #2886 (comment)), use the stream_data entry as is
  • otherwise, take the result of calling get_prep_value on the native block value.

[zxchew2014]

Check how can we solve it, as I also have this issue.
possible to give the situation