API slow when retrieving pipelines

[skrchnavy]

It is reported that UV is slow when retrieving list of pipelines (86 pipelines took more then 5 seconds).

See OpenDataNode/open-data-node#213 for more details

[skrchnavy]

If possible it shall be fixed in release branch to include into 2.2.0. If fix is too complex then via feature branch.

[eea03]

I investigated this issue and experienced similar problems, which were probably connected with memory issues.
I had about 20 pipelines in database and when running master API to obtain these pipelines, it took ~2 s.
When I added more pipelines (~40 in database) and tried to obtain them via master API, I got OutOfMemory error. This would cause a timeout in CKAN.

However when I added more memory to Tomcat, I had no performance issues at all. I added some more pipelines (~100 in database) and was able to obtain them via master API in ~30 ms.
Note that this was all on localhost.

I tried to test this also on eDemo DEV environment, where CKAN and UV are on separate servers, but I never got timeout. (UV has ~80 pipelines in database)

I could not find any performance issue in code either. Database query for getting pipelines is not optimal as it is JOIN of 3 tables but with such small tables it should not be an issue.

So conclusion: From what I tested I would suspect that this is related to memory issues with master component. To detect precisely where the memory issue is, it would require much more thorough testing.

As a workaround I would increase memory for Tomcat, I used this configuration and had no issue and everything worked quckly:

CATALINA_OPTS=-Dfile.encoding=UTF-8 -Xms512m -Xmx4096m -XX:PermSize=256m -XX:MaxPermSize=512m

[skrchnavy]

Probably same issue is #431