Extremely slow

[ivanelson]

This process via RPC is extremely slow! Or am I doing something wrong? My base has more than 14,000 tickets. The migration process is running for more than 8 hours and does not end.

[nazavode]

Hi, I guess that you're doing everything fine, unfortunately I've experienced very high latencies in RPC replies from Trac. You can try to run the command with the maximum verbosity level (-vvv) just to be informed about what's going on. Please let me know.

[ivanelson]

Hi,
Yeah! I will try according to your suggestion(-vvv).

If I export and then import. Is it faster?

[nazavode]

Unfortunately no, the slow guy is the export step via XML-RPC :) Let me know!

[ivanelson]

The process started around: 11:10am

And now that's in the Ticket: 582

2017-07-12 11:10:38,353 Trac[main] DEBUG: Dispatching <RequestWithSession "POST '/login/xmlrpc'">
2017-07-12 11:25:51,984 Trac[authz_policy] DEBUG: Checking TICKET_VIEW on ticket:582@*

[nazavode]

That is really too slow, let me run some tests on my Trac instance. In the meantime could you post a more beefy chunk of Trac log pls?

[ivanelson]

See the log in the dropbox: https://www.dropbox.com/s/1w4wl6vztci0sv5/trac.log?dl=0
Or gist: https://gist.github.com/ivanelson/629c8df330e6aadfd56b932da391f297

[nazavode]

Thanks. Your Trac log seems fine, the export process is actually retrieving all the ticket contents. Having a look at the full log, each ticket retrieval on the Trac side looks like this:

2017-07-12 11:53:48,349 Trac[xml_rpc] DEBUG: RPC(xml) 'ticket.get' result: ...
lot of permissions checking...
2017-07-12 11:53:49,923 Trac[xml_rpc] DEBUG: RPC(xml) 'ticket.changeLog' result: ...
lot of permissions checking...
2017-07-12 11:53:51,537 Trac[xml_rpc] DEBUG: RPC(xml) 'ticket.get' result: ...

So each ticket retrieval seems to keep Trac busy for about 3 to 4 seconds (variable depending on the size of the ticket's changelog), that leads to a total time of 14 hours needed to retrieve 13000 tickets!

[nazavode]

For huge instances like yours, the best solution would be to access directly the Trac database but I didn't think about that functionality yet.

[ivanelson]

Well! Maybe I can change the script and just select <> 'closed' tickets.

Decrease the universe! While I think of something in the future.

5 needs_work
21 testing
14530 closed
183 assigned
50 new
58 accepted
7 reopened

14.854

5 needs_work
21 testing
183 assigned
50 new
58 accepted
7 reopened

324 Tickets

[nazavode]

This looks like a good idea to me, closing this while opening a feature request in #30

[ivanelson]

Hi,

In conversation with the Trac community, I was advised to use multicall to improve performance.
See: https://trac-hacks.org/wiki/XmlRpcPlugin#/multicall

[nazavode]

Hi, that makes sense to me, anyway I'm more focused on correctness first, performance comes after the first usable release. Created a feature request in #31