File upload fails for custom script if filesize above ~2.5Mbyte

[dmulyalin]

Current Behavior

Was writing custom script and using FIleVar implemented file upload field in the form, however, trying to upload files larger then 2.5Mbyte fails with error giving traceback in container logs:

netbox-docker-netbox-1               | Internal Server Error: /extras/scripts/upload.Upload/
netbox-docker-netbox-1               | Traceback (most recent call last):
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner
netbox-docker-netbox-1               |     response = get_response(request)
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
netbox-docker-netbox-1               |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 84, in view
netbox-docker-netbox-1               |     return self.dispatch(request, *args, **kwargs)
netbox-docker-netbox-1               |   File "/opt/netbox/netbox/utilities/views.py", line 44, in dispatch
netbox-docker-netbox-1               |     return super().dispatch(request, *args, **kwargs)
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 119, in dispatch
netbox-docker-netbox-1               |     return handler(request, *args, **kwargs)
netbox-docker-netbox-1               |   File "/opt/netbox/netbox/extras/views.py", line 720, in post
netbox-docker-netbox-1               |     job_result = JobResult.enqueue_job(
netbox-docker-netbox-1               |   File "/opt/netbox/netbox/extras/models/models.py", line 565, in enqueue_job
netbox-docker-netbox-1               |     queue.enqueue(func, job_id=str(job_result.job_id), job_result=job_result, **kwargs)
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/rq/queue.py", line 514, in enqueue
netbox-docker-netbox-1               |     return self.enqueue_call(
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/django_rq/queues.py", line 68, in enqueue_call
netbox-docker-netbox-1               |     return self.original_enqueue_call(*args, **kwargs)
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/django_rq/queues.py", line 64, in original_enqueue_call
netbox-docker-netbox-1               |     return super(DjangoRQ, self).enqueue_call(*args, **kwargs)
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/rq/queue.py", line 410, in enqueue_call
netbox-docker-netbox-1               |     return self.enqueue_job(job, pipeline=pipeline, at_front=at_front)
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/rq/queue.py", line 572, in enqueue_job
netbox-docker-netbox-1               |     job.save(pipeline=pipe)
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/rq/job.py", line 694, in save
netbox-docker-netbox-1               |     mapping = self.to_dict(include_meta=include_meta)
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/rq/job.py", line 629, in to_dict
netbox-docker-netbox-1               |     'data': zlib.compress(self.data),
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/rq/job.py", line 305, in data
netbox-docker-netbox-1               |     self._data = self.serializer.dumps(job_tuple)
netbox-docker-netbox-1               | TypeError: cannot pickle '_io.BufferedRandom' object

Was trying to set these configuration parameters:

FILE_UPLOAD_MAX_MEMORY_SIZE=26214400
DATA_UPLOAD_MAX_MEMORY_SIZE=26214400

in these files:
confiugration/confiugration.py
enc/netbox.env
configuration/extra.py

but it seems as if configuration parameters ignored by Netbox or not supported.

Expected Behavior

Maximum upload size can be adjusted using these Django parameters on configuration.py file:

FILE_UPLOAD_MAX_MEMORY_SIZE=26214400
DATA_UPLOAD_MAX_MEMORY_SIZE=26214400

Docker Compose Version

Docker Compose version v2.12.2

Docker Version

Client: Docker Engine - Community
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:02:19 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 18:00:24 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.10
  GitCommit:        770bd0108c32f3fb5c73ae1264f7e503fe7b2661
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

The git Revision

N/A

The git Status

was runing from master branch, latest

Startup Command

docker-compose up

NetBox Logs

netbox-docker-netbox-1               | Internal Server Error: /extras/scripts/upload.Upload/
netbox-docker-netbox-1               | Traceback (most recent call last):
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner
netbox-docker-netbox-1               |     response = get_response(request)
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
netbox-docker-netbox-1               |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 84, in view
netbox-docker-netbox-1               |     return self.dispatch(request, *args, **kwargs)
netbox-docker-netbox-1               |   File "/opt/netbox/netbox/utilities/views.py", line 44, in dispatch
netbox-docker-netbox-1               |     return super().dispatch(request, *args, **kwargs)
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/django/views/generic/base.py", line 119, in dispatch
netbox-docker-netbox-1               |     return handler(request, *args, **kwargs)
netbox-docker-netbox-1               |   File "/opt/netbox/netbox/extras/views.py", line 720, in post
netbox-docker-netbox-1               |     job_result = JobResult.enqueue_job(
netbox-docker-netbox-1               |   File "/opt/netbox/netbox/extras/models/models.py", line 565, in enqueue_job
netbox-docker-netbox-1               |     queue.enqueue(func, job_id=str(job_result.job_id), job_result=job_result, **kwargs)
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/rq/queue.py", line 514, in enqueue
netbox-docker-netbox-1               |     return self.enqueue_call(
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/django_rq/queues.py", line 68, in enqueue_call
netbox-docker-netbox-1               |     return self.original_enqueue_call(*args, **kwargs)
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/django_rq/queues.py", line 64, in original_enqueue_call
netbox-docker-netbox-1               |     return super(DjangoRQ, self).enqueue_call(*args, **kwargs)
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/rq/queue.py", line 410, in enqueue_call
netbox-docker-netbox-1               |     return self.enqueue_job(job, pipeline=pipeline, at_front=at_front)
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/rq/queue.py", line 572, in enqueue_job
netbox-docker-netbox-1               |     job.save(pipeline=pipe)
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/rq/job.py", line 694, in save
netbox-docker-netbox-1               |     mapping = self.to_dict(include_meta=include_meta)
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/rq/job.py", line 629, in to_dict
netbox-docker-netbox-1               |     'data': zlib.compress(self.data),
netbox-docker-netbox-1               |   File "/opt/netbox/venv/lib/python3.10/site-packages/rq/job.py", line 305, in data
netbox-docker-netbox-1               |     self._data = self.serializer.dumps(job_tuple)
netbox-docker-netbox-1               | TypeError: cannot pickle '_io.BufferedRandom' object

Content of docker-compose.override.yml

version: '3.4'
services:
  netbox:
    ports:
      - 8000:8080
    build:
      context: .
      dockerfile: Dockerfile-Plugins
    image: netbox:latest-plugins
  netbox-worker:
    image: netbox:latest-plugins
    build:
      context: .
      dockerfile: Dockerfile-Plugins
  netbox-housekeeping:
    image: netbox:latest-plugins
    build:
      context: .
      dockerfile: Dockerfile-Plugins

[tobiasge]

I think the problem is the method the jobs are put into the queue. That's not something we can change in the Docker image.
@jeremystretch Could you have a look at the stack trace?

[kkthxbye-code]

FILE_UPLOAD_MAX_MEMORY_SIZE would need to be exposed as a setting in core netbox.

[dmulyalin]

@tobiasge and @kkthxbye-code thank you for your comments, am I understood you correct that I should move this request under the core netbox repository as a new feature request?

[tobiasge]

Yes, this should be a request in core NetBox.

[alexanderhofstaetter]

we should raise the default request size limit in docker/nginx-unit.json

Even though the NetBox core now properly exposes FILE_UPLOAD_MAX_MEMORY_SIZE, uploads larger than ~2.5 MB are still blocked in netbox-docker before they reach Django. The reason is the default NGINX Unit limit of roughly 2.5 MB (http.max_body_size).

To prevent 413 “Request Entity Too Large” errors for file uploads (custom scripts, image attachments, etc.), please consider a policy change in this repository:

Increase the default Unit request size limit to 100 MB (or higher) directly in the official config.

The change is minimal. In docker/nginx-unit.json add or merge the following block:

{
  "settings": {
    "http": {
      "max_body_size": 104857600
    }
  }
}

This ensures that all deployments based on netbox-docker can upload reasonably sized files out-of-the-box,
without having to patch the Unit configuration manually.

This would save future users from running into the 2.5 MB limit and aligns the container defaults with
the capabilities already present in the NetBox core.

Thanks for considering this as a default-policy enhancement for the docker image.