When json_encode() encounters a string that contains invalid utf8 characters it will return null.

[sevidmusic]

When json_encode() encounters a string that contains
invalid utf8 characters it will return null.

The solution is to use JSON_INVALID_UTF8_SUBSTITUTE to
convert invalid UTF-8 characters to \0xfffd.

@see https://www.php.net/manual/en/json.constants.php
@see https://stackoverflow.com/questions/4663743/how-to-keep-json-encode-from-dropping-strings-with-invalid-characters

The following integration test should pass (it currently fails):

<?php

/**
 * Purpose of this integration test:
 *
 * Test that strings that contain invalid utf8 characters can be
 * encoded as json via a Json instance, and that a Json instance
 * used to encode an strings that contain invalid utf8 character
 * can be decoded back to it's original value via a JsonDecoder.
 *
 */

include(
    str_replace(
        'tests' . DIRECTORY_SEPARATOR . 'integration',
        '',
        __DIR__
    ) .'vendor/autoload.php'
);

use \Darling\PHPJsonUtilities\classes\encoded\data\Json;
use \Darling\PHPJsonUtilities\classes\decoders\JsonDecoder;

/**
 * Return a string composed of a random number of randomly
 * generated characters.
 *
 * Note: The string may contain invalid utf-8 characters.
 *
 * @return string
 *
 * @example
 *
 * ```
 * echo randomChars();
 * // example output: rqEzm*g1vRI7!lz#-%q
 *
 * echo randomChars();
 * // example output: @Lz%R+bgR#79l!mz-
 *
 * ```
 *
 */
function randomChars(): string
{
    $string = str_shuffle(
        'abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*()_-=+'
    );
    try {
        $string .=
            random_bytes(random_int(1, 100)) .
            $string .
            random_bytes(random_int(1, 100));
    } catch(\Exception $e) {
    }
    return str_shuffle($string);
}

$string = randomChars();

$jsonEncodedString = new Json($string);

$expectedJsonString = json_encode(
    $string,
    JSON_INVALID_UTF8_SUBSTITUTE,
    2147483647
);

echo "\033[38;5;0m\033[48;5;111mRunning test" . __FILE__ . " \033[48;5;0m";

if(
    $jsonEncodedString->__toString() === $expectedJsonString
) {
    echo "\033[38;5;0m\033[48;5;84mPassed\033[48;5;0m";
} else {
    echo "\033[38;5;0m\033[48;5;196m Failed\033[48;5;0m" . PHP_EOL . PHP_EOL;
}

echo "\033[38;5;0m\033[48;5;45m Expected: \033[48;5;0m" . PHP_EOL . PHP_EOL;
var_dump($expectedJsonString);
echo "\033[38;5;0m\033[48;5;202m Actual: \033[48;5;0m" . PHP_EOL . PHP_EOL;
var_dump($jsonEncodedString->__toString());

Current result:

Expected: 

string(629) ""\u0007\ufffd\u000fau \ufffd\u000f\ufffd%y->k\ufffdu!3\ufffd\u001b6\ufffd&^(X*65jdt\ufffd\nt1-\ufffd\ufffd39\ufffd;$r\ufffdfs6j7o\ufffdg)-\ufffdymv\ufffd*@\u0000\ufffd<`\ufffd\n\ufffdr@\ufffd\u0006r!\u0001_\ufffduc\u0012%\ufffdh;\ufffd5d\ufffdpn^\u0005\ufffdc7\ufffdrq\ufffdi#\u0015\u0014\ufffd@,)9\ufffd)\ufffd+B\ufffd\ufffdmX%wa4_\ufffd\u0012z$b0LnVpz\ufffd8\ufffdlw\u0012.h\u001f\ufffd=(\ufffdqp$\ufffdh\ufffd6\ufffd\u07ce+\fvxe\ufffdo0a`)$l&\fgl\ufffd^nb\ufffd2\ufffd4\ufffd+\ufffdxj\ufffdJ_\ufffd8o+\ufffde\"\ufffd\ufffds#\ufffd\u001a\ufffd\ufffd<\ufffdx1A\ufffd\ufffdfo\ufffd=G\u05a0$\bg>#&W\ufffd25k\ufffdi5F4\ufffd\ufffd""
�[38;5;0m�[48;5;202m Actual: �[48;5;0m

string(0) ""

[sevidmusic]

As of 0dde4f8 the JSON_INVALID_UTF8_SUBSTITUTE flag is used which fixes the invalid characters.

However, this prevents strings that contain invalid chars from being encoded and decoded properly to their original value.

This needs further investigation.