kannel split messages

[travisghansen]

I recently updated to 0.9.9.2 and noticed the messages are getting sent to kannel in some sort of encoding. While the messages are getting sent with the proper text they are being split into multiple messages apparently based on 160 encoded characters.

Can you shed some light on how I can resolve this?

Note that I have sqlbox in the mix if that helps at all.

I also recently updated kannel to 1.5.0 but I've tested sending content to it directly and the # of messages are correct based on message length.

[antonraharja]

can you paste some real example ? like what text message you're trying to send, how long the text message etc.

anton

[travisghansen]

Sure

Example Message:

Info: Preparing Star::Root...
Info: Loading Star::Configuration from '../../../starbound.config'
Info: Done preparing Star::Root.
Info: Client version 'Beta

3 entries in sqlbox

0049006E0066006F003A00200050007200650070006100720069006E006700200053007400610072003A003A0052006F006F0074002E002E002E000D000A0049006E0066006F003A0020004C006F006100640069006E006700200053007400610072003A003A0043006F006E00660069006700750072006100740069006F006E002000660072006F006D0020

0027002E002E002F002E002E002F002E002E002F00730074006100720062006F0075006E0064002E0063006F006E0066006900670027000D000A0049006E0066006F003A00200044006F006E006500200070007200650070006100720069006E006700200053007400610072003A003A0052006F006F0074002E000D000A0049006E0066006F003A00200043

006C00690065006E0074002000760065007200730069006F006E002000270042006500740061

3 actual messages delivered to target phone:

1st ending with "Configuration from"
2nd ending with "Info: c"
3rd ending with "Beta"

Perhaps it's forcing the messages to be sent as unicode? I've not got the box checked..

[antonraharja]

Yes, its a unicode 00 49 00 6E.

Set logstate to 3 (default is 2) on config.php, and then watch the log:
tail -f /var/log/playsms/playsms.log | tee test.txt

Send again, watch the log, ctrl c when done. gather the test.txt, read it and edit it for private informations, and then send to my email antonrd@gmail.com (or if you're sure not private info, then paste it in pastebin.com and paste the pastebin url here for us to read)

[dvoryanchikov]

Travis, can u post here ur smsbox & sqlbox & kannel configuration, pw stripped, pls
especially max-messages & concatenation parameters in sendsms-user group for corresponding user from playsms gateway config.

[travisghansen]

@antonraharja in an effort to not post the full log with private info (and not go clear it all out manually) I did see this:

2014-01-01 10:12:53 PID52c44c959ab94 - L3 kannel outgoing # unicode autodetected

Does that help?

[travisghansen]

@dvoryanchikov I can but I've tested out kannel manually with longer messages and all works as configured. If you'd still like the configs I'll get them cleaned up and send over what you need.

[travisghansen]

So I just launched the url from playsms config manually and experienced the same behavior. I then removed the "&coding=2" parameter getting sent to kannel and it gets delivered as a single message. Is that parameter the result of unicode being auto detected?

[travisghansen]

I meant url from the playsms logs..not config :)

[antonraharja]

need a longer logs.

just tail it, do a single test and stop tailing, paste the log content here.

anton

[dvoryanchikov]

Travis, I know that when dealing with kannel through sqlbox u need max-messages set to more than one and concatenate set to true else u can get such behaviour when messages gets splitted. I guess u use one sendsms-user for test kannel and another one configured in playsms. Try to set max-messages=7 and concatenate=true for playsms user in kannel conf, restart boxes and test it.

[antonraharja]

playsms log would show whether or not the message originally submitted to Kannel as unicode, if it was then more entries on that log possibly could show why.

single SMS not unicode 160 chars, unicode 70.
concatenated SMS, not unicode 153 each, unicode 67 chars.

[travisghansen]

I'm certain it's not a kannel issue, I've run numerous tests through kannel using the same credentials/etc that playsms uses and everything works as desired. The culprit is the &coding=2 parameter getting sent to kannel via the http sendsms service. If I remove that option all works as it should.

I've cleansed sensitive data:

2014-01-01 10:12:52 PID52c44c9403c8a wm1 L2 sendsms # start uid:5 sender:000
2014-01-01 10:12:52 PID52c44c9403c8a wm1 L3 sendsms # footer: maxlength:459 msgcount:162 message:Info: Preparing Star::Root... Info: Loading Star::Configuration from '../../../starbound.config' Info: Done preparing Star::Root. Info: Client version 'Beta
2014-01-01 10:12:52 PID52c44c9403c8a wm1 L2 sendsms_queue_create # saving queue_code:caa0090052ce976fd3334ee7c558cff0 src:000
2014-01-01 10:12:52 PID52c44c9403c8a wm1 L2 sendsms_queue_create # saved queue_code:caa0090052ce976fd3334ee7c558cff0 id:7
2014-01-01 10:12:52 PID52c44c9403c8a wm1 L2 sendsms_queue_push # saving queue_code:caa0090052ce976fd3334ee7c558cff0 dst:9998884444
2014-01-01 10:12:52 PID52c44c9403c8a wm1 L2 sendsms_queue_push # saved queue_code:caa0090052ce976fd3334ee7c558cff0 smslog_id:1000063
2014-01-01 10:12:52 PID52c44c9403c8a wm1 L2 sendsms # end queue_code:caa0090052ce976fd3334ee7c558cff0 sms_count:1
2014-01-01 10:12:53 PID52c44c959ab94 - L2 sendsmsd # start processing queue_code:caa0090052ce976fd3334ee7c558cff0 sms_count:1 uid:5 gpid:0 sender_id:000
2014-01-01 10:12:53 PID52c44c959ab94 - L2 sendsmsd # sending queue_code:caa0090052ce976fd3334ee7c558cff0 smslog_id:1000063 to:9998884444 sms_count:1 counter:1
2014-01-01 10:12:53 PID52c44c959ab94 - L3 sendsms # dest number before manipulation: '9998884444'
2014-01-01 10:12:53 PID52c44c959ab94 - L3 sendsms # dest number after manipulation '9998884444'
2014-01-01 10:12:53 PID52c44c959ab94 - L3 msgtemplate # to:9998884444 msg:Info: Preparing Star::Root... Info: Loading Star::Configuration from '../../../starbound.config' Info: Done preparing Star::Root. Info: Client version 'Beta replacedby:Info: Preparing Star::Root... Info: Loading Star::Configuration from '../../../starbound.config' Info: Done preparing Star::Root. Info: Client version 'Beta
2014-01-01 10:12:53 PID52c44c959ab94 - L2 sendsms_process # start
2014-01-01 10:12:53 PID52c44c959ab94 - L2 simplerate getbyprefix # rate not found p_dst:9998884444
2014-01-01 10:12:53 PID52c44c959ab94 - L2 simplerate cansend # check username:wm1 sms_to:9998884444 credit:0.00 maxrate:0
2014-01-01 10:12:53 PID52c44c959ab94 - L2 simplerate cansend # allowed username:wm1 sms_to:9998884444 credit:0.00 maxrate:0
2014-01-01 10:12:53 PID52c44c959ab94 - L2 sendsms # saving smslog_id:1000063 u:5 g:0 gw:kannel s:000 d:9998884444 type:text unicode:0 status:0
2014-01-01 10:12:53 PID52c44c959ab94 - L2 sendsms_process # saved smslog_id:1000063 id:1067
2014-01-01 10:12:53 PID52c44c959ab94 - L3 sendsms # final smslog_id:1000063 message:Info: Preparing Star::Root... Info: Loading Star::Configuration from '../../../starbound.config' Info: Done preparing Star::Root. Info: Client version 'Beta len:162
2014-01-01 10:12:53 PID52c44c959ab94 - L3 kannel outgoing # start smslog_id:1000063 uid:5 to:9998884444
2014-01-01 10:12:53 PID52c44c959ab94 - L3 kannel outgoing # unicode autodetected
2014-01-01 10:12:53 PID52c44c959ab94 - L3 kannel outgoing # http://domain.com:13013/cgi-bin/sendsms?username=&password=&from=000&to=9998884444&dlr-mask=31&dlr-url=http%3A%2F%2Fdomain.com%2Findex.php%3Fapp%3Dcall%26cat%3Dgateway%26plugin%3Dkannel%26access%3Ddlr%26type%3D%25d%26smslog_id%3D1000063%26uid%3D5&charset=UTF-8&coding=2&account=wm1&text=Info%3A+Preparing+Star%3A%3ARoot...%0D%0AInfo%3A+Loading+Star%3A%3AConfiguration+from+%27..%2F..%2F..%2Fstarbound.config%27%0D%0AInfo%3A+Done+preparing+Star%3A%3ARoot.%0D%0AInfo%3A+Client+version+%27Beta&smsc=default
2014-01-01 10:12:53 PID52c44c959ab94 - L3 kannel outgoing # smslog_id:1000063 response:0: Accepted for delivery
2014-01-01 10:12:53 PID52c44c959ab94 - L3 dlr # isdlrd:0 smslog_id:1000063 p_status:0 uid:5
2014-01-01 10:12:53 PID52c44c959ab94 - L3 kannel outgoing # end smslog_id:1000063 p_status:0
2014-01-01 10:12:53 PID52c44c959ab94 - L2 simplerate deduct # enter smslog_id:1000063
2014-01-01 10:12:53 PID52c44c959ab94 - L2 simplerate getbyprefix # rate not found p_dst:9998884444
2014-01-01 10:12:53 PID52c44c959ab94 - L2 simplerate deduct # deduct smslog_id:1000063 msglen:159 count:1 rate:0 charge:0 credit:0.00 remaining:0
2014-01-01 10:12:53 PID52c44c959ab94 - L2 simplerate setusercredit # saving uid:5 remaining:0
2014-01-01 10:12:53 PID52c44c959ab94 - L2 sendsms_process # end
2014-01-01 10:12:53 PID52c44c959ab94 - L2 sendsmsd # result queue_code:caa0090052ce976fd3334ee7c558cff0 to:9998884444 flag:1 smslog_id:1000063
2014-01-01 10:12:53 PID52c44c959ab94 - L2 sendsmsd # finish processing queue_code:caa0090052ce976fd3334ee7c558cff0 uid:5 sender_id:000 sms_count:1
2014-01-01 10:12:58 PID52c44c9ab81d3 - L2 kannel call # start load:/var/www/domain.com/web/plugin/gateway/kannel/dlr.php
2014-01-01 10:12:58 PID52c44c9ab81d3 - L2 kannel dlr # addr:172.29.0.2 host:domain.com type:8 smslog_id:1000063 uid:5
2014-01-01 10:12:58 PID52c44c9ab81d3 - L3 dlr # isdlrd:0 smslog_id:1000063 p_status:1 uid:5
2014-01-01 10:12:58 PID52c44c9ab81d3 - L2 kannel call # end load dlr

[antonraharja]

I suspect something.

Edit your lib/fn_core.php, remove core_detect_unicode() in there and add this:

/**
 * Function: core_string_to_gsm()
 * This function encodes an UTF-8 string into GSM 03.38
 * Since UTF-8 is largely ASCII compatible, and GSM 03.38 is somewhat compatible, unnecessary conversions are removed.
 * Specials chars such as € can be encoded by using an escape char \x1B in front of a backwards compatible (similar) char.
 * UTF-8 chars which doesn't have a GSM 03.38 equivalent is replaced with a question mark.
 * UTF-8 continuation bytes (\x08-\xBF) are replaced when encountered in their valid places, but
 * any continuation bytes outside of a valid UTF-8 sequence is not processed.
 * Based on https://github.com/onlinecity/php-smpp
 *
 * @param string $string
 * @return string
 */
function core_string_to_gsm($string)
{
    $dict = array(
        '@' => "\x00", '£' => "\x01", '$' => "\x02", '¥' => "\x03", 'è' => "\x04", 'é' => "\x05", 'ù' => "\x06", 'ì' => "\x07", 'ò' => "\x08", 'Ç' => "\x09", 'Ø' => "\x0B", 'ø' => "\x0C", 'Å' => "\x0E", 'å' => "\x0F",
        'Δ' => "\x10", '_' => "\x11", 'Φ' => "\x12", 'Γ' => "\x13", 'Λ' => "\x14", 'Ω' => "\x15", 'Π' => "\x16", 'Ψ' => "\x17", 'Σ' => "\x18", 'Θ' => "\x19", 'Ξ' => "\x1A", 'Æ' => "\x1C", 'æ' => "\x1D", 'ß' => "\x1E", 'É' => "\x1F",
        // all \x2? removed
        // all \x3? removed
        // all \x4? removed
        'Ä' => "\x5B", 'Ö' => "\x5C", 'Ñ' => "\x5D", 'Ü' => "\x5E", '§' => "\x5F",
        '¿' => "\x60",
        'ä' => "\x7B", 'ö' => "\x7C", 'ñ' => "\x7D", 'ü' => "\x7E", 'à' => "\x7F",
        '^' => "\x1B\x14", '{' => "\x1B\x28", '}' => "\x1B\x29", '\\' => "\x1B\x2F", '[' => "\x1B\x3C", '~' => "\x1B\x3D", ']' => "\x1B\x3E", '|' => "\x1B\x40", '€' => "\x1B\x65"
    );
    $converted = strtr($string, $dict);
    return $converted;
}

/**
 * Function: core_detect_unicode()
 * This function returns an boolean indicating if string needs to be converted to utf
 *  to be send as an SMS
 * @param $text
 *      string to check
 * @return int unicode
 */
function core_detect_unicode($text) {
        $unicode = 0;
    $textgsm=core_string_to_gsm($text);

    $match=preg_match_all('/([\\xC0-\\xDF].)|([\\xE0-\\xEF]..)|([\\xF0-\\xFF]...)/m',$textgsm,$matches);
    if ($match!==FALSE) {
        if ($match==0) {
            $unicode = 0;
        } else {
            $unicode = 1;
        }
    } else {
        //TODO broken regexp in this case, warn user
    }
        return $unicode;
}

And also check your plugin/gateway/kannel/fn.php, around line 56. It should be updated to:

//Automatically setting the unicode flag if necessary
        if (!$unicode)
                $unicode=core_detect_unicode($sms_msg);

In master repo the code to check unicode are updated. I hope that will fix this issue.

[antonraharja]

Or if its a new install, perhaps you want to consider upgrade with 1.0-beta2

[dvoryanchikov]

I beleive splitting messages is not unicode issue. Sure, if send this test message in 7bit (coding=0) it should not be splitted. But what about messages longer than 160 chars. I bet it is smsbox issue, known in my installation, I solved it with max-messages & concatenate parameters.

[travisghansen]

I tried updating to 1.0-beta2 and after some nasty debugging due to silent failure (@code is bad) I'm getting this:

Fatal error: Cannot redeclare _PEAR_call_destructors() (previously declared in /var/www/domain.com/web/lib/external/pear-db/PEAR.php:735) in /usr/share/php/PEAR.php on line 777

This is after removing the @ sign in DB::connect for include_once "DB/${type}.php"; in the else block.

@dvoryanchikov yes splitting messages up that are longer than 160 is expected and has worked for me for years. I have max-messages up very high for the playsms account in kannel. The issue here is splitting up messages at < 160 chars.

[travisghansen]

Here is the fix for the PEAR issue:

diff --git a/web/init.php b/web/init.php
index 5045841..df62997 100644
--- a/web/init.php
+++ b/web/init.php
@@ -119,6 +119,8 @@ define('HTTP_PATH_TPL', $http_path['tpl']);
$core_config['apps_path'] = $apps_path;
$core_config['http_path'] = $http_path;

+set_include_path($apps_path['libs'] . '/external/pear-db' . PATH_SEPARATOR . get_include_path());
+
// load init functions
include_once $apps_path['libs']."/fn_init.php";

[travisghansen]

OK, after getting PEAR issue worked out unicode detection appears to be fixed and a single message is getting sent. A couple notes:

1. The unicode autodetected log message is no longer in the log
2. coding=2 is omitted from the http request to kannel

Unfortunately, via the web interface my phonebook is no longer working..fortunately the entries are still in the database. Any ideas on how to get that working with beta2?

[dvoryanchikov]

Travis, smsbox can send long messages in one SMS, use concatenate = 1 for this.
Anton, 3dr person experiencing troubles after upgrade to 1.0, please take a look at my comments da3213a

[dvoryanchikov]

Travis, since 1.0 phonebook contact can be in several groups, u should fill new table playsms_toolsPhonebook_group_contacts with ur phonebook ids and ids of any group u have

[travisghansen]

Also not seeing my "Incoming Messages" for this account.

[travisghansen]

Is there an upgrade script/path for filling that table? I do have some contacts that are duplicates to be in multiple groups so that would be nice to get those cleaned up.

[dvoryanchikov]

There is no any migrate script until u write one and share with us :-)
Try to find incoming messages in new "Unhandled messages" menu or "All incoming" in admin account.

[travisghansen]

@dvoryanchikov my kannel setup is very simple connected to a usb phone. The phone will not send out > 160 at a time. So yes, kannel is sending to the phone in 1 transaction but the phone is splitting them appropriately.

[dvoryanchikov]

Travis, any gsm phone/modem CAN send long (multipart) messages , at least all of my phones and modems can.

[travisghansen]

Messages aren't visible in "All incoming" and I don't see "Unhandled messages" Where can I find it?

WRT to the messages thanks for the clarification but in my case it doesn't really matter. The point is that somewhere further down the line (out of my control) is splitting the messages (carrier to carrier, etc). I expect messages > 160 ultimately reaching my target phone to be split into multiple messages.

[dvoryanchikov]

Unhandled should be in Administration->Sandbox
You also can set max messages option in playSMS so you can control amount of parts sent at time.

[travisghansen]

Yeah, the messages I used to see for this account are not visible in Sandbox either :(

I just noticed the DROP gpid in the upgrade SQL. I could help with upgrade SQL but DROP'ing that is absolutely a no-no until the upgrade has occurred.

The phonebook listing IMO should show contacts even if they aren't associated to a group.

[travisghansen]

Scratch the incoming messages problem. I was looking in the wrong spot :( Sorry for the mixup.

[dvoryanchikov]

I tried to implement that feature (make contacts free of groups). This is possible, but conflicts next accepted features "shared groups and shared contacts". I'm currently working on it..

[travisghansen]

@dvoryanchikov excellent!

[antonraharja]

so, has the original issue fixed in 1.0-beta2 ?

anton

[travisghansen]

@antonraharja yes, no longer sending coding=2 and messages go out in segments of 160 real chars as before. Original issue can be closed.