max_document_size check is not accurate

[nickva]

max_document_size currently checks document sizes based on Erlang's external term size of the jiffy-decoded document body. This makes sense because that's what used to store the data on disk and it's what manipulated by the CouchDB internals.

However erlang term size is not always a good approximation of the size of json encoded data. Sometimes it can be way off (I've seen 30% off) and It's hard for users to estimate or check the external term size beforehand. So for example if max_document_size is 1MB, CouchDB might reject user's 600KB json document because Erlang's external term size of that document greater than 1MB.

Possible Solutions

• Do nothing. If the current size check is to throttle or limit disk usage and disk usage is driven by the external size (though in most cases compression is used on it, so it will usually be less), then on that level it makes sense to keep it as is.

• Re-encode the data using jiffy and check the size against that. That's a better check but will impact performance. However this is also not an exact solution. Users' json encoder might insert more whitespace (say as indentation), or whitespace after commas, use a different algorithm for encoding floating point numbers (scientific notation, represent exact floating point numbers without a decimal point (5 instead of 5.0 etc.). So the size would still be off. Made a PR with this approach: Provide a more accurate size check for max_document_size limit #660

• Like above but enhance jiffy to return an "encoded size" without actually doing the encoding. Or least have it do the encoding internally and just return byte size to Erlang instead of a full term which will have to be thrown away.

• Here is at attempt to do a size check in Erlang. Since jiffy is pretty quick is might end being slower than doing the full encoding in jiffy:

encsize({[]}) ->
    2;  % opening { and closing }

encsize({Props}) ->
    % 2 is because opening { and closing }
    % Inside the lc 2 is for : and , but counts an extra , at end
    % -1 is to subtract last , from lc part
    2 + lists:sum([encsize(K) + encsize(V) + 2 || {K, V} <- Props]) - 1;

encsize([]) ->
    2;  % opening [ and closing ]

encsize(List) when is_list(List) ->
    % 2 is for [ and ]
    % inside the lc 1 is for , but it counts one extra , for last element
    2 + lists:sum([encsize(V) + 1 || V <- List]) - 1;

encsize(Float) when is_float(Float) ->
    % this doesn't match what jiffy does. so will be off
    byte_size(float_to_binary(Float, [{decimals, 16}, compact]));

encsize(Integer) when is_integer(Integer), Integer >= 0 ->
    trunc(math:log10(Integer)) + 1;

encsize(Integer) when is_integer(Integer), Integer < 0 ->
    % 2 is because 1 is for the - character
    trunc(math:log10(Integer)) + 2;

encsize(Binary) when is_binary(Binary) ->
    % count escapes and assume they'd be escaped with one extra escape character
    Escapes = [<<"\b">>, <<"\f">>, <<"\n">>, <<"\r">>, <<"\t">>, <<"\\">>,
        <<"\"">>],
    MatchCount = length(binary:matches(Binary, Escapes)),
    % 2 is for open and closing "
    2 + byte_size(Binary) + MatchCount;

encsize(Atom) when is_atom(Atom) ->
    encsize(atom_to_binary(Atom, utf8)).

Maybe modify it to provide an always underestimating check. So users get the benefit of the doubt. That is whatever encoding they pick, find a not too expensive check that will be accurate enough and always be smaller?

[iilyak]

@nickva: There is an implementation in erlang here https://github.com/okeuday/erlang_term/blob/master/src/erlang_term.erl. MIT license.

[iilyak]

Another implementation could be based on https://gist.github.com/iilyak/a11a481bd3f7311d8499e19f5e4c8f22

[nickva]

We don't really need it be that generalized in a way because we know we'd only get what jiffy decoded which is object proplists {[...]}, arrays (lists [...]) and then integers, floats and binary (but unicode and escape sequences have to be accounted for). Oh maybe atoms but those are just translated to binary and handled that way.

[iilyak]

Another option is to change chttpd:json_body to return {Size, Body} and pass Size through.

[nickva]

@iilyak
Interesting idea. It would depend how much the code would be affected _bulk_docs and possibly multipart/related PUTs might be an issue too.

[nickva]

This was fixed. Close it.