Bug when submitting answers with non-latin characters when using XMLRPC

[taniwallach]

#956 added support for rendering UTF-8 problems via xmlrpc, in particular changes to clients/sendXMLRPC.pl and lib/WebworkClient.pm and the formats in the `lib/WebworkClient/' directory.

However, if an answer (or the value of any form field) includes non-latin1 characters, things break.

Some ideas of what may be causing the problem appear in the comments to #956 but finding the root cause and fixing it remains to be done.

The same issue forced leaving the values of the 3 buttons (submit, preview, show correct answers) in English.

[taniwallach]

I just ran into a similar problem with a different feature which used XMLRPC.

If I have a problem using AnswerHints as a post-filter, and the message being set uses non-Latin1 characters, things work fine when the problem is operated "normally" but the problem fails to render in the instructor/sets2/setName context when the "Render problem" icon is clicked. (That feature uses an XMLRPC call to renderProblem.)

After a short delay, the browser opens a popup with the text:

webwork2.technion.ac.il says
/webwork2/instructorXMLHandler: Forbidden

and the Apache error.log file get lines like:

Error message for command: renderProblem
<br/>faultcode: Client
<br/>faultstring: Wide character in subroutine entry at /usr/share/perl5/XMLRPC/Lite.pm line 181.

<br/>End error message<br/>
[Thu Aug 22 14:47:45.036034 2019] [perl:error] [pid 1628] [client <IP-NUMBER>:50384] [/webwork2/instructorXMLHandler] Error message for command: renderProblem \n<br/>faultcode: Client \n<br/>faultstring: Wide character in subroutine entry at /usr/share/perl5/XMLRPC/Lite.pm line 181.\n \n<br/>End error message<br/>\n * in WeBWorK::ContentGenerator::instructorXMLHandler::content called at line 233 of /opt/webwork/webwork2/lib/WeBWorK/ContentGenerator.pm\n * in WeBWorK::ContentGenerator::go called at line 384 of /opt/webwork/webwork2/lib/WeBWorK.pm, referer: https://webwork2.technion.ac.il/webwork2/<COURSE_NAME>/instructor/sets2/xmlRPwideCharacterIssue/?key=<KEY>&effectiveUser=<eUser>&user=<User>

Below is a sample PG file (cut down to be relatively small) file which triggers this "new" problem on the current WeBWorK 2.15 pre-release branch:

DOCUMENT();

loadMacros(
  "PGstandard.pl",
  "MathObjects.pl",
  "answerHints.pl"
);

TEXT(beginproblem());

Context("Numeric");

Context()->texStrings;
BEGIN_TEXT
Enter \( x^2 \) as your answer: \{ans_rule(10)\}
END_TEXT
Context()->normalStrings;


ANS(Compute("x^2")->cmp->withPostFilter(AnswerHints(Compute("2*x") => "אבג")) 
);

ENDDOCUMENT();

By replacing the Hebrew letters "אבג" with "ABC" the error will be bypassed.

[taniwallach]

I suspect the problem is really occurring in the renderProblem subroutine of lib/WebworkWebservice/RenderProblem.pm when $out2 is being defined. That anonymous hash has 3 fields

• text
• header_text
• WARNINGS
  which are sent through encode_utf8_base64 but other data in this hash can apparently contain Unicode text which is not being encoded into UTF-8 (and then base64). It seems that if any "wide characters" remain inside the data, then XMLRPC::Lite cannot handle it.

Based on what is shown for "AnswerGroupInfo" it seems that in the example file above the "AnswerHints" post-filter apparently installs something containing the Unicode message string in the AnswerEvaluator hash inside the post_filters array. Unfortunately, the output just shows "too deep" so I cannot confirm this suspicion.

[taniwallach]

I seem to have found a very small patch to /usr/share/perl5/XMLRPC/Lite.pm with fixes both these problems totally. It involved "updating" XMLRPC::Lite to leave UTF-8 encoded text "alone" and not encode it into base64. As modern standards for XMLRPC no longer only allow ASCII text, this is now considered valid.

The patch replaces line 55 of /usr/share/perl5/XMLRPC/Lite.pm which originally was:

     base64 => [10, sub {$_[0] =~ /[^\x09\x0a\x0d\x20-\x7f]/}, 'as_base64'],

with the following replacement line:

     base64 => [10, sub {$_[0] =~ /[^\x09\x0a\x0d\x20-\x7f]/ && !utf8::is_utf8($_[0])}, 'as_base64'],

It is possible that after this change, the manual encoding to base64 after UTF-8 for 3 specific values in $out2 of lib/WebworkWebservice/RenderProblem.pm is no longer needed (not the decode on the other side).

The patch above is based on several posts/pages on the internet about problems with XMLRPC::Lite and SOAP::Lite and Unicode:

• https://www.perlmonks.org/?node_id=955106
• https://www.perlmonks.org/?abspart=1;displaytype=displaycode;part=2;node_id=1154371
• http://www.london.pm/pipermail/london.pm/Week-of-Mon-20061023/005171.html
• http://blog.livedoor.jp/dankogai/archives/50375819.html (pay attention to the code, and ignore the Japanese text)

[taniwallach]

@mgage @heiderich - Can you see if you can hit the same issues above before patching /usr/share/perl5/XMLRPC/Lite.pm and if the patch fixes the problems also for you. For Docker, I mounted the modified version of the file from outside the image. If it helps, I will add it to the provided files, and try to have docker-compose.yml mount it.

[nopuk]

Hello,
I seem to have stumbled across this/similar issue and am hoping for some guidance. Have webwork installed, however when I go to Library browser to either add a new homework, or try to select a math category such as "calculus single variable" or try to add a local problem every time I encounter the error message:
/webwork2/instructorXMLHandler: Forbidden

I tried the above fix for line 55 in Lite.pm, it did nothing for me.

Can anyone help?

[taniwallach]

The XMLRPC::Lite issue is only when Unicode characters are making problems. There are several other causes of problems with the library browser.

1. Did you run OPL-update? (See below.)
2. Is your server configured to connect the correct way (http/https same as in regularly in use, correct host name, etc.)
3. Look for more details inside the Apache log files inside the running Docker container.

Note: It would be faster to try the code in #1002 which loads prepared OPL tables than to really run OPL-update.

[nopuk]

Hello again,
So I 'solved my problem' purely by pot luck, went into the webwork and changed - /opt/webwork/webwork2/conf/site.conf

$server_root_url = 'http://what my url actuially is.ie/';
it previously localhost I think, putting in the IP address did nothing but the URL seems to have worked?

Just wanted to post the comment in case anyone else happens upon this ( as you can tell I know zero about network stuff)

[drgrice1]

This is resolved with the removal of XMLRPC!