New lite-server implementation

[tiran]

The new development server depends on werkzeug instead of paste. The
werkzeug WSGI server comes with some additional features, most
noticeable multi-processing server. The IPA framework is not compatible
with threaded servers. Werkzeug can serve static files easily and has a
fast auto-reloader.

The new lite-server implementation depends on PR #314 (privilege
separation). For Python 3 support, it additionally depends on PR #393.

Signed-off-by: Christian Heimes cheimes@redhat.com

[pvoborni]

Shouldn't we rather remove lite sever completely?

[rcritten]

I always found the lite-server to be incredibly helpful for server-side plugin development. If it isn't being used any more then I'd wonder what is being used instead.

[martbab]

@rcritten AFAIK most of us develop plugins locally, then build RPMs which we sync to a VM, install IPA server and use ipa-run-tests to test the new plugin. I personally used lite-server only once or twice when I started out with my contribution.

We had some discussion earlier in which we concluded that lite-server is not used anymore and should be removed (this is probably what @pvoborni is referring to). If you can provide compelling arguments against this decision then we can reconsider.

[rcritten]

Right, and IMHO that development process is inefficient and prone to error. Rather than copying bits around and doing full installs over and over you can run the server in-tree and have vastly improved debugging available.

Certainly a "final" test of a full server install loop is necessary but for initial development and testing the lite-server is far easier and efficient. At one time the tests were also run almost exclusively in-tree (which was faster at the time).

[pvoborni]

When I was still developing server-side then I used lsyncd to rsync files from my local laptop working git repo to their location on vm in a lab.

So the process for me was just:

• change file
• restart httpd
• test

Lite sever covers only API changes. Solution above covered also installs and updates (to some extend). So it was much more usable because API changes is only a small part of FreeIPA development.

For Web UI, there is older /install/ui/util/sync.sh which does similar thing.

Build system refactoring enabled to use make install method

$ mkdir /tmp/vm
$ sshfs -o transform_symlinks root@<vm>:/ /tmp/vm
$ make install DESTDIR=/tmp/vm

This covers all use cases. So it might be better to talk if we should rather promote this method with e.g. containerized IPA instance.

I.e. Before we ACK or NACK this PR. I'd rather have a conversation, what is the issue and what is the right solution. How we can make the whole process better. And then update http://www.freeipa.org/page/Contribute/Code which is rather obsolete and doesn't describe any method.

[tiran]

I put some effort into the dev server because I find it very useful during development. The werkzeug WSGI adds some useful features, e.g. auto-reloader and through-the-web debugger (soon). I can maintain a copy of the dev server for me personally but I'd rather have a copy in the source tree. It doesn't have to be /lite-server.py. How about contrib/dev-server?

[pvoborni]

Maybe the lite-server approach is great and other people would appreciate that. I shouldn't be the one to judge it.

What about demonstrating it to the rest of the team and showing the value?

[tiran]

PR #314 has landed. I have rebased the branch and made the lite-server even more convenient to use. You can now run it with make lite-server or make lite-server PYTHON=python3. It tells you how to set up a Kerberos ccache, too.

With the help of the lite-server, I found issue pyldap/pyldap#84 within ten seconds.

Now with WSGI profiler!

[tiran]

Example of a single request profile with new lite-server:

127.0.0.1 - - [15/Feb/2017 12:55:20] "POST /ipa/session/json HTTP/1.1" 200 -
ipa: INFO: [jsonserver_session] admin@IPA.EXAMPLE: json_metadata(None, None, command=u'all', version=u'2.218'): SUCCESS
--------------------------------------------------------------------------------
PATH: '/session/json'
         6551240 function calls (4596653 primitive calls) in 1.869 seconds

   Ordered by: internal time, call count
   List reduced from 436 to 30 due to restriction <30>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
2013370/405370    0.303    0.000    0.462    0.000 /usr/lib64/python2.7/json/encoder.py:341(_iterencode_dict)
  1755304    0.278    0.000    0.278    0.000 {isinstance}
     3556    0.201    0.000    0.446    0.000 /home/heimes/redhat/freeipa/ipalib/parameters.py:441(__init__)
187490/446    0.150    0.000    0.999    0.002 /home/heimes/redhat/freeipa/ipalib/util.py:58(json_serialize)
 110038/1    0.127    0.000    0.236    0.236 /home/heimes/redhat/freeipa/ipalib/rpc.py:277(json_encode_binary)
     3999    0.085    0.000    0.256    0.000 /home/heimes/redhat/freeipa/ipalib/parameters.py:954(__json__)
   173558    0.075    0.000    0.075    0.000 {hasattr}
440062/395518    0.072    0.000    0.239    0.000 /usr/lib64/python2.7/json/encoder.py:288(_iterencode_list)
   405370    0.057    0.000    0.520    0.000 /usr/lib64/python2.7/json/encoder.py:417(_iterencode)
143774/143772    0.052    0.000    0.052    0.000 /home/heimes/redhat/freeipa/ipalib/base.py:123(__setattr__)
   104200    0.036    0.000    0.070    0.000 {setattr}
        1    0.035    0.035    0.560    0.560 /usr/lib64/python2.7/json/encoder.py:186(encode)
        1    0.029    0.029    0.029    0.029 {built-in method sasl_interactive_bind_s}
   234842    0.026    0.000    0.026    0.000 {getattr}
 4445/446    0.025    0.000    0.461    0.001 /home/heimes/redhat/freeipa/ipalib/util.py:62(<dictcomp>)
     4449    0.024    0.000    0.032    0.000 {sorted}
   234451    0.019    0.000    0.019    0.000 {method 'get' of 'dict' objects}
        1    0.018    0.018    0.018    0.018 {method 'encode' of 'str' objects}
   133044    0.016    0.000    0.016    0.000 {_json.encode_basestring_ascii}
    24961    0.011    0.000    0.011    0.000 {_codecs.utf_8_decode}
    24961    0.011    0.000    0.030    0.000 {method 'decode' of 'str' objects}
   107753    0.010    0.000    0.010    0.000 /home/heimes/redhat/freeipa/ipalib/parameters.py:506(<genexpr>)
     3556    0.010    0.000    0.010    0.000 /home/heimes/redhat/freeipa/ipalib/parameters.py:261(parse_param_spec)
    13348    0.008    0.000    0.008    0.000 {method 'items' of 'dict' objects}
     7176    0.008    0.000    0.012    0.000 /home/heimes/redhat/freeipa/ipalib/text.py:248(as_unicode)
    91841    0.007    0.000    0.007    0.000 /usr/lib64/python2.7/json/encoder.py:361(<lambda>)
    24961    0.007    0.000    0.018    0.000 /usr/lib64/python2.7/encodings/utf_8.py:15(decode)
     1973    0.007    0.000    0.274    0.000 /home/heimes/redhat/freeipa/ipalib/parameters.py:725(clone_retype)
     7388    0.006    0.000    0.006    0.000 {method 'match' of '_sre.SRE_Pattern' objects}
    77523    0.006    0.000    0.006    0.000 {method 'pop' of 'dict' objects}

[MartinBasti]

Fixed upstream
master:
https://fedorahosted.org/freeipa/changeset/ff6e701b0077d9c8e2aacdcaecf70f885018db92