Don't ignore nomultipart when storage is low

[CarstenGrohmann]

When the temporary storage filled up, the old implementation uploaded all data with multipart uploads, even if "nomultipart" was set.

The new implementation emits a warning and returns -ENOSPC instead.

Fixes #1595

Exampe:

$ cp -p 3000M /s3/mybucket/
cp: error writing "/s3/mybucket/3000M": No space left on device
cp: failed to extend "/s3/mybucket/3000M": No space left on device

New syslog output w/ warning:

2021-03-08T19:12:16.609323+01:00 mysrv <user.warning> s3fs[4842]:fdcache_entity.cpp:Write(1485): Not enough local storage to cache write request: [path=/3000M][fd=6][offset=1835008][size=65536]
2021-03-08T19:12:16.609522+01:00 mysrv <user.warning> s3fs[4842]:s3fs.cpp:s3fs_write(2310): failed to write file(/3000M). result=-28

New debug output w/ warning:

unique: 35, opcode: WRITE (16), nodeid: 2, insize: 65616, pid: 3759
write[5] 65536 bytes to 1835008 flags: 0x8001
2021-03-08T19:23:37.509Z [DBG] s3fs.cpp:s3fs_write(2298): [path=/3000M][size=65536][offset=1835008][fd=5]
2021-03-08T19:23:37.509Z [DBG] fdcache.cpp:ExistOpen(526): [path=/3000M][fd=5][ignore_existfd=false]
2021-03-08T19:23:37.509Z [DBG] fdcache.cpp:Open(446): [path=/3000M][size=-1][time=-1]
2021-03-08T19:23:37.509Z [DBG] fdcache_entity.cpp:Dup(246): [path=/3000M][fd=5][refcnt=2]
2021-03-08T19:23:37.509Z [DBG] fdcache_entity.cpp:Open(315): [path=/3000M][fd=5][size=-1][time=-1]
2021-03-08T19:23:37.509Z [DBG] fdcache_entity.cpp:Dup(246): [path=/3000M][fd=5][refcnt=3]
2021-03-08T19:23:37.509Z [DBG] fdcache_entity.cpp:Close(200): [path=/3000M][fd=5][refcnt=2]
2021-03-08T19:23:37.509Z [DBG] fdcache_entity.cpp:Write(1439): [path=/3000M][fd=5][offset=1835008][size=65536]
2021-03-08T19:23:37.509Z [WAN] fdcache_entity.cpp:Write(1485): Not enough local storage to cache write request: [path=/3000M][fd=5][offset=1835008][size=65536]
2021-03-08T19:23:37.509Z [WAN] s3fs.cpp:s3fs_write(2310): failed to write file(/3000M). result=-28
2021-03-08T19:23:37.509Z [DBG] fdcache.cpp:Close(593): [ent->file=/3000M][ent->fd=5]
2021-03-08T19:23:37.509Z [DBG] fdcache_entity.cpp:Close(200): [path=/3000M][fd=5][refcnt=1]
   unique: 35, error: -28 (No space left on device), outsize: 16

[gaul]

Thank you for your contribution @CarstenGrohmann! I am suspicious of the ensure_diskfree logic and there may be other bugs here.