huginn acts very slow, how can i shorten the time

[biguan99]

my scenario have a webhook agent, two java script agent (with name js1 and js2) and a post agent.
i also create a request client, who send 12 or more request to webhook agent. js1 receives event from webhook agent, calculates time consumed from the time this request make, then passes to js2. js2 calculates time consumed from the time this request make, then passes to post agent.
i alse create a server to get the post request from my post agent, who can calculate the time consumed.

code of request client:

var async = require('async');
var request = require('request');

function makeRequest(cb){
        var now = new Date().getTime();
        var uri = 'http://xxxxxxxxx/users/1/web_requests/11/123';
        request.post(uri, {
                json : {
                        time : now,
                }
        }, function(err, res){
                console.log('diff ' + (new Date().getTime() - now));
                cb();
        });
}

async.timesLimit(12, 3, function(n, next){
        makeRequest(next);
}, function(){
        console.log('mission done');
        setTimeout(function(){
                process.exit();
        }, 50);
});

------ end ------

the result:
send time: 1542351548663, diff1 : 539, diff2 : 1110, diff : 2041
send time: 1542351548656, diff1 : 768, diff2 : 1745, diff : 3406
send time: 1542351548665, diff1 : 1498, diff2 : 2564, diff : 4549
send time: 1542351549350, diff1 : 1660, diff2 : 2934, diff : 4879
send time: 1542351549758, diff1 : 1554, diff2 : 3538, diff : 6041
send time: 1542351549927, diff1 : 2218, diff2 : 4330, diff : 7650
send time: 1542351550696, diff1 : 1821, diff2 : 4063, diff : 7423
send time: 1542351551097, diff1 : 2335, diff2 : 4825, diff : 9139
send time: 1542351551299, diff1 : 2436, diff2 : 5351, diff : 10084
send time: 1542351551929, diff1 : 2779, diff2 : 6484, diff : 11634
send time: 1542351552092, diff1 : 2980, diff2 : 6469, diff : 11949
send time: 1542351552879, diff1 : 2797, diff2 : 6567, diff : 12446
send time: 1542351553078, diff1 : 3245, diff2 : 7573, diff : 13995
send time: 1542351553534, diff1 : 3227, diff2 : 8066, diff : 15321
send time: 1542351553723, diff1 : 3749, diff2 : 8791, diff : 16610
send time: 1542351554147, diff1 : 3463, diff2 : 8497, diff : 16728
send time: 1542351554569, diff1 : 4076, diff2 : 9824, diff : 18772
send time: 1542351554986, diff1 : 4339, diff2 : 10377, diff : 19866
send time: 1542351555158, diff1 : 4462, diff2 : 10736, diff : 20218
send time: 1542351555842, diff1 : 4586, diff2 : 11078, diff : 20889

as you can see, i received my result after a very time, how can i shorten the time?

[biguan99]

web: bundle exec unicorn -c config/unicorn.rb
schedule: bundle exec rails runner bin/schedule.rb
dj: bundle exec script/delayed_job run
dj2: bundle exec script/delayed_job -i 2 run
dj3: bundle exec script/delayed_job -i 3 run
#dj4: bundle exec script/delayed_job -i 4 run
#dj5: bundle exec script/delayed_job -i 5 run
#dj6: bundle exec script/delayed_job -i 6 run

[dsander]

That are the time logged as diff1, diff2 and diff?

The setting that affects the processing time the most is propagate immediately. With it checked in the Agent configuration, events received will be send to the receiver directly. The default setting is to put the received events in a queue and process them every minute.

You can also tweak DELAYED_JOB_SLEEP_DELAY, the value determines how long the backgrounds workers are sleeping between attempts to fetch new jobs from the database, you could go as low as 0.1 seconds but that will increase the (idle) load on the database.

Are the clocks of your server and client in sync?

How many CPU cores does the Huginn server have? With the configuration you listed it should have least 8-10 cores. If you have fewer I would reduce the amount of delayed job processes. You need one for the database, two for the unicorn webserver and then one of each delayed job process.

[biguan99]

i had js1, js2 and post agent checked for propagate immediately. webhook agent has no propagate setting.
when i made those post requests, the cpu load is 20%, memory is 34%.

[biguan99]

my huginn server has 2 cpu cores and 4g mem. as for scenarios like mine, how fast can i get my post agent request?

[dsander]

I think changing DELAYED_JOB_SLEEP_DELAY in your .env file or docker environment configuration will make the biggest change. With only 2 cpus I don't thing having that many delayed job processes helps. If you have a lot of Website or PostAgents that are waiting for external services increasing the amount of delayed job process count can increase the throughput.