Skip to content

Conversation

netzkr
Copy link

@netzkr netzkr commented Apr 11, 2022

#33636 , New Feature for bulk create/update

With ModelBulkProcessMixin , We could minimize memory usage.
Without ModelBulkProcessMixin, We shold maintain bulk array size up to 100_000 or manually maintain arraysize up to batch_size like 10_000

`

    class BulkyModel(models.Model, ModelBulkProcessMixin):
        id = models.BigAutoField(primary_key=True)
        name = models.CharField(max_length=30, null=False)

    names = [f"name-{num}" for num in range(100_100_000)]

    # Case : Raw bulk_create
    objs = [BulkyModel(name=name) for name in names]
    BulkyModel.objects.bulk_create(
        objs
    )  # We should maintain big array size and this leades to OOM error

    # Case : Chunked bulk_create
    objs = list()
    for name in names:
        obj = BulkyModel(name=name)
        objs.append(obj)
        if len(objs) > 10_1000:
            BulkyModel.objects.bulk_create(objs)
            objs.clear()

    if len(objs) > 0:
        BulkyModel.objects.bulk_create(objs)
        objs.clear()

    # Case : With ModelBulkProcessMixin
    with BulkyModel.gen_bulk_create(batch_size=10_000) as bulk:
        for name in names:
            bulk.add(BulkyModel(name=name))`

@timgraham
Copy link
Member

Closing per ticket.

@timgraham timgraham closed this Apr 11, 2022
nessita added a commit to dtraleigh/django that referenced this pull request Oct 26, 2023
…fixtures docs.

Also, added details about loading multiple fixtures and unified line wrapping
at 79 cols.

Co-Authored-By: Aniketh Babu <anikethbabu@gmail.com>
Co-Authored-by: Mariusz Felisiak <felisiak.mariusz@gmail.com>
Co-Authored-By: Natalia Bidart <124304+nessita@users.noreply.github.com>
nessita pushed a commit that referenced this pull request Oct 27, 2023
…e fixtures docs.

Also, added details about loading multiple fixtures and unified line wrapping
at 79 cols.

Co-Authored-By: Aniketh Babu <anikethbabu@gmail.com>
Co-Authored-by: Mariusz Felisiak <felisiak.mariusz@gmail.com>
Co-Authored-By: Natalia Bidart <124304+nessita@users.noreply.github.com>

Backport of 334dc07 from main
nessita pushed a commit that referenced this pull request Oct 27, 2023
…e fixtures docs.

Also, added details about loading multiple fixtures and unified line wrapping
at 79 cols.

Co-Authored-By: Aniketh Babu <anikethbabu@gmail.com>
Co-Authored-by: Mariusz Felisiak <felisiak.mariusz@gmail.com>
Co-Authored-By: Natalia Bidart <124304+nessita@users.noreply.github.com>

Backport of 334dc07 from main
nessita added a commit to nessita/django that referenced this pull request Oct 30, 2023
nessita added a commit that referenced this pull request Oct 30, 2023
nessita added a commit that referenced this pull request Oct 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants