Add AVMaxFileSize config option

[VicDeo]

Description

• Adds File size limit
• Fixes fwrite(): send of 1028 bytes failed with errno=32 Broken pipe at .... #124 and Antivirus data stream problem fills log file #125

Changes in Docs:

Stream Length (bytes) 26214400 (25Mb) by default. Should be <= than StreamMaxLength ClamAV value
File size limit, -1 means no limit (bytes) -1 by default - prevent background scanner from checking large files (Has no effect while scanning newly uploaded files)

To test:

1. set StreamMaxLength in /etc/clamd.conf to e.g. 10M
2. Configure the app in daemon or daemon(socket) mode
3. cat several 100Mb files with EICAR signature
4. Try upload or background scan

[mention-bot]

@VicDeo, thanks for your PR! By analyzing the history of the files in this pull request, we identified @FCTURNER, @DeepDiver1975 and @vgezer to be potential reviewers.

[VicDeo]

@PVince81 please have a look

[PVince81]

unrelated, but reminded me of a potential issue, raised as #137

[PVince81]

Code looks good.

@owncloud/qa can you help testing ?

@VicDeo Any chance to add unit tests of some sorts ? That quite some complex logic there.
Even a test that only covers the stream stuff using a dummy stream would be nice.

For the docs change, please raise a ticket in the documentation repo.

[PVince81]

@VicDeo question: what if the virus signature overlaps with the retry-boundary, can it still be detected properly ?

[PVince81]

maybe the unit test could have its own dummy signature and have the mock stream receiver detect it (which simulates whatever clamd does)

[VicDeo]

@PVince81

what if the virus signature overlaps with the retry-boundary, can it still be detected properly ?

E.g. When Stream Length contains 3 chunks. And file has 7.
The scanning will be done as
1st scan request: chunks 1-2-3
2nd scan request: chunks 3-4-5
3rd scan request: chunks 5-6-7

if a signature is spread across chunks 2-3-4 - the answer is No
In general ClamAV doesn't support large files at all. So it is still an improvement :)

Hm. I've just came upon an idea to hide chunk size from UI and make it completely internal.

[ms270169]

@VicDeo, you are right (I was wrong), byteCount ist cleared to zero in initScanner(). Virus scanning works now as desired.

I tested it with the following configuration:

• testfile: 30000 bytes length, 68 byte EICAR virus signature on position 16360-17027
• config-value Chunk Size: 8192 (config not used, because I add the file with plus button)
• used chunk-size: 8192
• config-value Stream Length: 20000
• config-value File size limit: -1

Virus signature overlaps second and third chunk. Virus is only detected if second chunk is repeated after reinitialisation of stream (if block overlapping is implemented).

Result:
Virus detection successful, File not accepted by the cloud-system.

The user gets the info "Unsupported media Type", which is a confusing message. If the other bug, leading to this wrong message could not be fixed (see #125 (comment)), I would recommend to add a log entry in case of virus detection (for example before avirwrapper.php#L99).

[VicDeo]

@ms270169
Filesystem InvalidContentExeption is mapped to HTTP status 415.
"Unsupported media Type" is a text representation for HTTP status 415.
There is no special HTTP status "User just uploaded a virus" unfortunately.

I can't do anything better on the antivirus side. 403 with message "Forbidden" is neither too much descriptive. :/

[VicDeo]

grr. It drive me nuts. All methods of Config mock return null on the second test execution.

[ms270169]

@VicDeo , maybe you have an idea how to better visualize the message. Because the information about a detected virus is inside the HTTP 415 response. If the message of the response would be shown, everything would be clear for the user...

<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>OCA\DAV\Connector\Sabre\Exception\UnsupportedMediaType</s:exception>
  <s:message>Virus Eicar-Test-Signature is detected in the file. Upload cannot be completed.</s:message>
</d:error>

[VicDeo]

I still have no idea why PHPUnit mocks return null for the second testcase.
I created own mocks instead of mocking with PHPUnit API.
It works this way.

[VicDeo]

@PVince81 ready from my POV

[PVince81]

@VicDeo from what I see this was tested and only needs a code review, correct ?

[VicDeo]

@PVince81 yes
@ms270169 helped me a lot with a design overview.

[PVince81]

The code looks good, great to have tests!

@VicDeo I didn't find where "AVMaxFileSize" is covered / passed in the tests, mind pointing me there ? I guess it probably has another name

[VicDeo]

@PVince81 Not covered.
It is used in background scanner only, because file size is unpredictable while using a stream wrapper.
BackgroundScanner::getFilesForScan is the only place in code where is it used.

[PVince81]

Ok, fine 👍

[VicDeo]

Stable9.1: #140
Stable9: #144
Stable8.2: #145

[zubanst]

Any back-port plan for 9.0.x? In 9.0.6 I have the same issue described in #125 .

[VicDeo]

@zubanst would you like to test #144 ?

[zubanst]

@VicDeo I tested it on 9.0.6 and it works. I don't get anymore the log file filled. I tested also the reaction online trying to upload an EICAR file and it detects it and stops the upload. For me the fix can be applied in 9.0.6

[SergioBertolinSG]

Updated steps:

To reproduce:

1. set StreamMaxLength in /etc/clamd.conf to e.g. 10M
2. Configure the app in daemon or daemon(socket) mode
3. In files_antivirus config in administration change the values:
   Stream Length   to 10000000 bytes
   File size limit, -1 means no limit  to 10000000 bytes
4. cat several 100Mb files with EICAR signature
5. Try upload or background scan

Expected: It detects the virus.
Not expected: It uploads the file.

[VicDeo]

@SergioBertolinSG
Indeed Stream Length in files_antivirus configuration should match Clamav StreamMaxLength option.
File size limit has no effect on upload.