Add https to core system

[aragilar]

This adds some HTTPS support to NGAS. The main changes are whether sendfile is used (it's possible to use sendfile with https, but you need a recent linux kernel, very new openssl etc., so it makes sense to wait until that has stabilised before trying to add that), and using urls where possible to keep around the protocols. Let me know if there are any issues with this (I don't think there are any breaking changes.

Closes #8.

[rtobar]

Thanks a lot for this contribution @aragilar. It looks like very good work! I will go thoroughly through the code but that will probably have to wait until tomorrow or early next week, as my plate is currently a bit full.

I gave the patch a quick overview though, and I wanted to have a clear idea of the intent behind this.

• Initially I thought your idea was to have a setup with NGAS A pushing data through the subscription mechanism via HTTPS to a non-NGAS, proxy server B, which would authenticate the request, and forward the data to an NGAS server C via HTTP. Is that still the case? or maybe I misunderstood?
• If that scenario is still valid, then I'd refrain from adding HTTPS support to the NGAS server code, at least as part of this PR, as adding support to the client side is the only requirement.
• In the same spirit, and if the only reason for adding HTTPS support to the server was to test subscription via HTTPS, I'd replace the NGAS server with a dummy HTTPS server that consumes all incoming request data and drops it. That would remove the need for a fully-fledged NGAS server with HTTPS support just for the purpose of testing the subscription.
• I make this distinction between client and server code because although the HTTPS support you added on the NGAS server looks fine, is (I think) a small piece of the cake, and much further testing would need to be in place to make sure this is working fine across the different uses cases an NGAS server supports. It still looks like a good starting point, but I'd prefer to handle it separately from this PR if possible.
• A final more generic point: if you could break the changes into smaller, self-contained commits that would be great (if too difficult that's fine too); I personally find that a history with smaller atoms makes changes easier to reason about.

Please let me know your thoughts on this, specially about your use case as I might have misunderstood it from the beginning. I would definitely want to accept the client-side code changes as part of this PR as they are definitely useful! The server-side changes are the only thing that I need some clarifications on, basically to understand how necessary they are to your use case.

[coveralls]

Coverage decreased (-0.4%) to 68.383% when pulling 6d5f9a5 on aragilar:add-https into bbc56d5 on ICRAR:master.

[rtobar]

Regarding failures: the coverage always fluctuates a bit (it's not 100% deterministic), so it can be somewhat safely ignored. The MacOS build failed because it took too long -- and in general all jobs took longer than previously, do you have an idea of why this might be? For example:

$> git co master
$> pytest test/commands/test_containers.py::ngamsContainerTest::test_AppendRemove -v |& grep passed
=========================== 1 passed in 1.92 seconds ===========================
$> git co add-https
$> pytest test/commands/test_containers.py::ngamsContainerTest::test_AppendRemove -v |& grep passed
========================== 1 passed in 11.43 seconds ===========================

[rtobar]

@aragilar I just finished reviewing more in depth your changes. They look awesome! There are a few minor formatting things, and a few observations on the import machinery, but otherwise looks fine.

I think my reading yesterday was correct, in that you are adding https support on the server just to support testing. If that is the case, I'd rather include only the client-side changes in this PR, and replace the test with a dummy server. The server-side changes are still quite good, but I'd save them until we really start picking up https support on the server.

[aragilar]

I think the slowdown might be some of the TLS stuff, I'll see if I can speed it up.

[rtobar]

I think the slowdown might be some of the TLS stuff, I'll see if I can speed it up.

I think it was the time.sleep(10) right after starting up the server in the tests

[aragilar]

I think I've made all the changes. Major things:

• I've dropped all the pytest things, and moved it to the test helper file.
• I've tried getting the subscription notifier to work, but it's still failing (no idea why).
• I've left the tls server support in (but I've put it behind a check whether we can import ssl), as it managed to catch the sendfile issues, so keeping it around is still useful (I wouldn't use it in production though, there would need to be support for specifying tls versions and cypher suites and all that, and we'd probably want to split out the certificate from the private key, none of which are worth worrying about if we just use it to test).

[aragilar]

It looks like the mac build failed to install berkeleydb for some reason?

[rtobar]

I think this is still preventing the tests from running in Travis because NGAS_TESTDB is always set, but sometimes to an empty value; hence the coverage still doesn't show the code paths being executed. This will need to be changed to os.environ.get('NGAS_TESTDB', None) or similar.

[rtobar]

Also the if ssl is not None could be part of the condition within skipIf so the whole method won't require an extra level of indentation

[rtobar]

Instead of creating a configuration file, saving it, and passing it via cfgFile you can pass down a list of 2-element tuples directly to prepExtSrv via cfgProps and it will do that for you. Shouldn't change the behavior in any way, just make it easier to read the test code.

[rtobar]

I think this is causing problems when running the tests multiple times. Each time this get executed, a new certificate is appended into the vault for the same CA; then I get SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)') probably due to the different certificates for the same CA? If I go to $VIRTUAL_ENV/lib/python3.6/site-packages/certifi/cacert.pem and remove the last certificates generated by this code I can run the test again.

I think a better approach would be not to save the CA perm into the central vault, and instead capture it and give it to the ngamsPClient for usage with requests. It's a bit more of code, but you'd avoid touching the user's vault, and add the flexibility to the ngamsPClient to use arbitrary CA certs.

[rtobar]

Please see my comment above, I think the new UseSendfile setting is unnecessary given we have already ngasServer.get_server_access_proto or ngasServer._cert to distinguish this.

[rtobar]

Thanks for the updates! It looks neater now. The MacOS build seems to be a homebrew problem totally unrelated to us; I wouldn't worry about it for now, but I'm still giving it a couple of restarts to see if I it's intermittent enough so it goes away after a couple of tries.

I think at the moment the main points to solve are two: first, the new test is still not being executed in Travis, because of the condition used for skipping, please see my comments on that. That explains why the coverage still doesn't pick up the new code paths. And secondly, the notification listener check on the https subscription test fails because there's effectively no file transmission going through! This is because you are starting two server independently, and this is currently not well supported by the testing code (the second will overwrite the root directory created by the first). I'd suggest you do exactly what is done in the test_basic_subscription test: starts the two servers A and B in the same cluster (i.e., connected to the same database, and using the new support you are adding in this PR for being able to specify a cert_file in ngamsTestSuite.prepCluster), and then create a subscription from A to B. You should then be able to check that all works as planned. I'd even try to share most of the code of both tests.

[rtobar]

Hi @aragilar any news on this? I'm really looking forward to include these changes, which seem to be almost ready for inclusion. A few comments on the last push (there were are a couple of tests failing in Travis):

• The use of trustme looks great and now tests can be re-run with confidence. The extra new code at the end of ngamsTestLib.py is unnecessary now I believe? That'd include some of the new top-level imports too of course.
• The preparation of the subscription test I think could do something like this (with the small, corresponding change in the definition of _prep_subscription_cluster):

-                first_server = [[8778, []],]
-                second_server = [
-                    [8779, [
-                        ['NgamsCfg.ArchiveHandling[1].EventHandlerPlugIn[1].Name',
-                        'test.ngamsTestLib.SenderHandler'],
-                    ]],
-                ]
-                self.prepCluster(first_server, cert_file=cert_file)
-                self.prepCluster(second_server, cert_file=cert_file)
+                self._prep_subscription_cluster(8778, (8779, [], True), cert_file=cert_file)

• You would then want to assert for version 2 of the file instead of version 1.
• That also means you don't need two disconnected NGAS servers for this test. So you could simplify the declaration of the test_https_subscription test like this:

-    if ssl is not None:
-        @unittest.skipIf(os.environ.get('NGAS_TESTDB'), "Need two cluster for this test to run")
-        def test_https_subscription(self):
+    @unittest.skipIf(ssl is None, "Need SSL for this test to run")
+    def test_https_subscription(self):

• I still think the new UserSendfile preference is unnecessary, it should be possible to use the server's get_server_access_proto method instead.

I think that's the gist of it. Let me know if I can be of any assistance! :)

[aragilar]

Sorry, was working on other tasks today, I should have something either today or (more likely, given the time here) tomorrow. I know I tried running them as a single cluster previously, but had issues with the http vs https protocol things, I'll see if I modify the database whether that will be fixed.

[aragilar]

Apparently I didn't need to modify the database. I thought the cluster needed to communicate over http, but apparently not, I suspect the errors I was seeing previously when using a single cluster were down to the sendfile usage.

[rtobar]

Great! This looks even neater than before, and all checks are now happily passing. I also tested locally a couple of times and it all seems in working order, so I merged just now.

Thanks again @aragilar for the great effort! :)