client fails to upload very large files

[lzaoral]

If you try to update very large files (+2 GB) using kobo's upload_file XML-RPC method, the method will fail with the following Traceback:

$ truncate -s 2G test
$ osh/client/osh-cli mock-build --tarball-build-script=test ./test
Traceback (most recent call last):
  File "/src/osh/client/osh-cli", line 79, in <module>
    main()
  File "/src/osh/client/osh-cli", line 72, in main
    parser.run()
  File "/src/kobo/kobo/cli.py", line 296, in run
    cmd.run(*cmd_args, **cmd_kwargs)
  File "/src/osh/client/commands/cmd_diff_build.py", line 157, in run
    target_dir, self.parser)
  File "/src/osh/client/commands/shortcuts.py", line 145, in upload_file
    return hub.upload_file(os.path.expanduser(srpm), target_dir)
  File "/src/kobo/kobo/client/__init__.py", line 473, in upload_file
    upload_id, upload_key = self.upload.register_upload(os.path.basename(file_name), checksum, fsize, target_dir)
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 1112, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 1446, in __request
    allow_none=self.__allow_none).encode(self.__encoding, 'xmlcharrefreplace')
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 971, in dumps
    data = m.dumps(params)
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 502, in dumps
    dump(v, write)
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 524, in __dump
    f(self, value, write)
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 540, in dump_long
    raise OverflowError("int exceeds XML-RPC limits")
OverflowError: int exceeds XML-RPC limits

OpenScanHub needs to be able to work with large SRPMs. Unfortunately, I have no idea how to fix this issue without breaking backwards compatibility with older releases of kobo.

[rohanpm]

It seems there are at least two layers which are going to have a problem with this:

• XML-RPC client, as seen above.
• the DB, as upload objects have a field declared as size = models.PositiveIntegerField(). According to https://docs.djangoproject.com/en/4.2/ref/models/fields/#positiveintegerfield this is also 32-bit, having a max value of 2147483647.

[kdudka]

The former could be solved by introducing a new XML-RPC method. IIRC checksum is already passed in as a string rather than a long number. The latter could be handled by a Django migration.

[lzaoral]

I was able to come up with a solution that should not break anything that was not already broken: #219