Xmlrpc utf8 fix

[taniwallach]

Code to solve problems XMLRPC had with UTF-8 in some places.

See: the html2xml part of the report at openwebwork/pg#443 which was resolved by this change.

The main change is to encode UTF-8 data which before it gets sent to XMLRPC and decode it on arrival.

Problems which did not work over html2xml / XMLRPC before this change are now working, and no adverse issues have been seen in several weeks of use (over 8000 scores earned over html2xml using the LTI modifications) on a server running this code to get the html2xml system working with problems several of which did not load over html2xml without this patch.

[taniwallach]

openwebwork/pg#443 provides a sample of the errors received before the fix.

Below is a shortt PG file which triggers the issue before the fix:

DOCUMENT();        # This should be the first executable line in the problem.

loadMacros(
  "PGstandard.pl",
  "MathObjects.pl",
  "parserPopUp.pl"
);

SET_PROBLEM_TEXTDIRECTION("rtl");
SET_PROBLEM_LANGUAGE("he-IL");

TEXT(beginproblem());

Context("Numeric");
$popup1 = PopUp(['נכון', 'לא נכון'], 'נכון');

BEGIN_TEXT
\{$popup1->menu\}
END_TEXT

ANS($popup1->cmp);

ENDDOCUMENT();        # This should be the last executable line in the problem.

Avoiding the use of Hebrew in the Popup would avoid the issue before the patch in the PR. The PR fixes the issue.

< $popup1 = PopUp(['CORRECT', 'INCORRECT'], 'CORRECT');
---
> $popup1 = PopUp(['נכון', 'לא נכון'], 'נכון');

[taniwallach]

openwebwork/pg#443 provides a sample of the errors received before the fix.

Below is a shortt PG file which triggers the issue before the fix:

DOCUMENT();        # This should be the first executable line in the problem.

loadMacros(
  "PGstandard.pl",
  "MathObjects.pl",
  "parserPopUp.pl"
);

SET_PROBLEM_TEXTDIRECTION("rtl");
SET_PROBLEM_LANGUAGE("he-IL");

TEXT(beginproblem());

Context("Numeric");
$popup1 = PopUp(['נכון', 'לא נכון'], 'נכון');

BEGIN_TEXT
\{$popup1->menu\}
END_TEXT

ANS($popup1->cmp);

ENDDOCUMENT();        # This should be the last executable line in the problem.

Avoiding the use of Hebrew in the Popup would avoid the issue before the patch in the PR. The PR fixes the issue.

< $popup1 = PopUp(['CORRECT', 'INCORRECT'], 'CORRECT');
---
> $popup1 = PopUp(['נכון', 'לא נכון'], 'נכון');

[mgage]

Installing this PR removes the initial errors,

Unable to decode problem text
Error message for command: renderProblem
faultcode: Client
faultstring: Wide character in subroutine entry at /Volumes/WW_test/opt/local/lib/perl5/vendor_perl/5.12.3/XMLRPC/Lite.pm line 181.
End error message

but I still get the following error on the example problem.
Line 89 ismy $md5 = md5_hex($tex);

Michaels-MacBook-Pro-2:t mgage1$ wwperl ../sendXMLRPC.pl -bB xmlrpc_utf8_fix.pg
home directory ..
Wide character in subroutine entry at /Volumes/WW_test/opt/webwork/webwork2/../pg//lib/WeBWorK/EquationCache.pm line 89.

How should this be handled?

[mgage]

Using use Encode qw(encode_utf8, decode_utf8) and Replacing line 89 by
my $md5 = md5_hex(encode_utf8($tex));
seems to work but I don't know if that is the right fix.
@taniwallach is there a reason you don't see this error?

[mgage]

I'm pretty sure my $md5 = md5_hex(encode_utf8($tex)); is the right fix. See the bottom of this description of md5_hex: https://perldoc.perl.org/Digest/MD5.html

[mgage]

I'm going to pull this request and then submit a patch for the file
/pg/lib/WeBWorK/EquationCache.pm

[taniwallach]

Using use Encode qw(encode_utf8, decode_utf8) and Replacing line 89 by
my $md5 = md5_hex(encode_utf8($tex));
seems to work but I don't know if that is the right fix.
@taniwallach is there a reason you don't see this error?

@dpvc in openwebwork/pg#443 (comment) suggested a very similar solution to the EquationCache.pm issue. Using MathJax instead of images mode avoids that issue, and as I was focused primarily on solving the XMLRPC issue, I did not get back to checking the EquationCache.pm fix.