File carves are not accurately reporting errors

[zhumo]

Fleet version: 4.16

Operating system: N/A

Web browser: N/A

---

🧑‍💻  Expected behavior

When a file carve fails, Fleet should accurately report a failure. It should confirm that the file exists in the storage destination.

💥  Actual behavior

A file carve was set up with incorrect settings. When a file carve runs, however, the query still reports success.

This may only occur with the S3 carving backend. But please verify it for Fleet carving backend as well.

More info

Internal only: https://docs.google.com/document/d/15Y_86nzS4-9SklXnLpcYBrGYTxbvMHwgrozoZRxfjWA/edit?usp=sharing
^ Notes with detailed report in Oct 6, 2022 section

The key reproduction steps:

1. Set --carver_block_size (osquery flag) to a value that is smaller than the minimum block size for s3 multipart uploads (5MB). Customer set --carver_block_size=2000000.
2. Try to carve a file that is larger than carver_block_size.
3. Fleet will get an error from S3 when it tries to upload the 2MiB block but it still ends up showing a 100% completion status for the carve.

Whoever develops this fix will likely need access to an AWS sandbox subaccount for testing.

We don't need to support carving when the carver block size is smaller than the S3 minimum size, we just need to accurately return an error status so that we don't mistakenly give the user the impression the carve succeeded.

[zwass]

@zhumo thank you for filing the issue. I updated with further details on how to reproduce. It's actually an issue on the Fleet server, so I changed up the tags an assigned to @chiiph.

Whoever works on this issue can reach out to me if you need help getting things set up properly to reproduce.

[roperzh]

@zwass I was able to repro using minio, so fortunately no need to deal with AWS.

It seems like the server is doing all it can to let osquery know the request failed, from the logs, we return a 400 error on the last carve block:

level=error ts=2022-10-07T05:18:01.251864Z component=http user=unauthenticated method=POST uri=/api/v1/osquery/carve/block took=88.197083ms err="save block data: s3 multipart carve upload: EntityTooSmall: Your proposed upload is smaller than the minimum allowed object size.\n\tstatus code: 400, request id: 171BB1B5B66308CD, host id: "

By looking at osquery's source code seems like it always stores the status as SUCCESS and just logs errors

    // TODO: Error sending files.
    status = contRequest.call(params);
    if (!status.ok()) {
      VLOG(1) << "Post of carved block " << i
              << " failed: " << status.getMessage();
      continue;
    }
  }

  updateCarveValue(carveGuid_, "status", kCarverStatusSuccess);
  return Status::success();

Having said that, I think that Fleet should be able to do better, because even if osquery sets the right status, finding that the request failed it's a bit obscure as you have to run a SELECT * FROM carves and find your carve in the results, how to solve this sounds like a challenge for the product folks, is that correct?

[zwass]

Is it possible for us to make sure the fleetctl get carves output doesn't show completion as 100% in these cases? Even better, how hard would it be to show the actual error that occurred?

It seems like there's definitely some work to be done on the osquery side to improve how it handles errors, but I think we've got enough info on the Fleet side to be more clear as well?

[roperzh]

gotcha! I didn't think of that (I need to familiarize myself more with this), seems like it should be possible with minor adjustments, as we're not relying on osquery to tell us the completion state:

fleet/cmd/fleetctl/get.go

Lines 725 to 728 in b14c7af

	completion := fmt.Sprintf(
	"%d%%",
	int64((float64(c.MaxBlock+1)/float64(c.BlockCount))*100),
	)