Floating Point Support for Obliv-C

[darioncassel]

Purpose

Floating point support has been added to Obliv-C as an "obliv float".

Changes Made

Nearly every file related to the Obliv-C CIL extension has been changed. The most important changes were to:
- src/ext/processObliv/processObliv.ml
- src/ext/processObliv/oblivUtils.ml
- src/cil.ml
- src/ext/oblivc/obliv_bits.c
- src/ext/oblivc/obliv_bits.h
- src/ext/oblivc/obliv_common.h
- src/ext/oblivc/obliv_bits.c
- src/ext/oblivc/obliv_types_internal.h
- src/ext/oblivc/obliv_types.h
- src/ext/oblivc/obliv.h

Documentation

Comments have been added throughout the additions where appropriate to explain the changes. A guide to extending the language has been added to the README.

Testing

The tests/testFloat folder contains tests for each standard function used with obliv floats (plus, minus, multiplication, etc). The tests/linreg folder contains an implementation of linear regression performed with the obliv float data type.

[darioncassel]

@samee I've gone over the changes you've made and I believe that I've resolved the issues that were brought up either by you or by Jack on this thread. I've cherry-picked the latests commits from your float-support branch in order to absorb your changes into my PR, and on top of those I've made a few commits that fix the other issues. Please let me know how you want this rebased / modified before it can be merged into the main repo as I've tried rebasing it but it doesn't seem to come out cleanly. I'll be trying rebasing again after you get a chance to review these latest changes.

Responses to the points you brought up:

SCDtoObliv/SCDtoObliv.py

+class util_cppReadOp:
Is anybody using this class, or should this be removed?

You're right, that class is no longer necessary so I have removed it.

src/cil.ml

+ (* TODO: add something here for obliv floats *)
Can you quickly test what actually happens here? I don't want to merge this if we are silently returning the wrong size in sizeof(obliv float). If it's just 32-bits of OblivBits, I think you can just change the "isOblivSimple t" guard condition to accept "isOblivSimple t || isOblivFloat t". The rest of the code should just work.

I believe that you're correct about the required change. I have changed the guard isOblivSimple t to isOblivSimple t || isOblivFloat t since it is just 32 OblivBits throughout.

src/ext/oblivc/obliv.h

+void execFloatProtocol(ProtocolDesc* pd, protocol_run start, void* arg);
Looks like a temporary test function of some sort, one that nobody is using right now. Is this useful for future development, or should I remove it?

That function is for my FloatProtocol which I was using for testing during the development of the floating point support. I found it useful then but now you're right in saying that it doesn't have that much value. You can remove it and the FloatProtocol file if you'd like.

In __obliv_c__setEqualToF:

+ __obliv_c__assignBitKnown(dest,0);
why is this line necessary? Same question for other comparisons. I'm guessing you have it here because it was crashing on manual tests due to these being uninitialized. But in C code generated from Obliv-C, obliv bools are always initialized to zero anyway.

I'm not sure why it's necessary but the reason that I have it is because all of the comparisons (the ones that were there before I started adding floating point support) have it. For example __obliv_c__setNotEqual has __obliv_c__assignBitKnown(dest,0);. Should these lines be removed globally or am I missing something?

+      // Use the "union trick" to convert wd to a proper float
+      // since wd is stored as a widest_t
+      union ItoF {
+        widest_t i;
+        float f;
+      };
+      union ItoF conv;
+      conv.i = wd;
+      *dest = conv.f;

I think you could also replace all of this with *dest = (float)(&wd);. What we really want is to use some sort of an endian-neutral wrapper, but that's for a later fix.

I tried out that replacement and it doesn't seem to work. I actually tried out a few things before resorting to using the "union trick" but all of them did not work. What are you thoughts on this?

| Set(dv,CastE(dt,Lval sv),loc) when isOblivInt dt ->
    begin match unrollType dt,unrollType (typeOfLval sv) with
    | TInt(dk,da), TInt(sk,sa) when hasOblivAttr sa ->
        if isSigned sk then
          setIntExtend "__obliv_c__setSignExtend" dv dk sv sk loc
        else setIntExtend "__obliv_c__setZeroExtend" dv dk sv sk loc
    | _ -> instr
    end

Are you sure you don't need a TFloat case here? E.g. what happens when you do obliv float x = my_int; Or vice-versa? Or, you may also need to support conversions such as obliv float x = 5.0l;.

I'm not sure -- are we trying to support that kind of behavior? I can add in support for that but I hadn't because I didn't realize that we would want those kinds of conversions.

src/frontc/cabs2cil.ml

+
+    | TFloat _, TPtr _ -> result
+    | TPtr _, TFloat _ -> result

This looks out of place. Why is this necessary?

You're right, that has now been removed.

test/testFloat/

Will anything break if I move this to test/oblivc/testFloat/ ?

Things would have broken because of relative paths in the Makefile but I've fixed that and moved both testFloat and the linreg test to test/oblivc (and changed their Makefiles to make use of the common Makefile in test/oblivc/common).

test/testFloat/dbg.h

Looks like a duplicate file. Please include from a single place.

I've now included that file from a single place.

Other Changes

src/ext/processObliv/processObliv.ml

While I was testing out these changes at first compilation was not terminating on my test cases. I determined the cause to be the removal of | TFloat(k,_) -> fst fnames from

let setComparisonUS fnames dest s1 s2 loc = 
  let optype = typeOfLval s1 in
  let fname = match unrollType optype with
  | TInt(k,_) -> if isSigned k then snd fnames else fst fnames
  | _ -> E.s (E.error "Cannot operate on obliv values of type %a" d_type optype)
  in
  setComparison fname dest s1 s2 loc

which was done during the changes in your rebase. Adding back that case allows compilation to terminate and the test cases to run successfully. What was the reason for the removal?

src/ext/oblivc/obliv_bits.c

Moved spec.src_f = v from feedOblivInputs to the setup function:

void setupOblivFloat(OblivInputs* spec, __obliv_c__float* dest, float v)
  { 
    __obliv_c__setupOblivBits(spec,dest->bits,v,__bitsize(v));
    spec->src_f = v;
  }

As I mentioned earlier, I believe that this was the cause of issues when using the changes in this PR with other code. After this change, running:

git clone git@github.com:samee/sqrtOram.git
cd sqrtOram
export OBLIVC_PATH=path/to/oblivc
make build/bench/benchSqrtAccess
build/bench/benchSqrtAccess -- 1234 100 10 &
build/bench/benchSqrtAccess localhost 1234 100 10

no longer causes a segfault.

Misc.

The (*CHANGE*) markers have been removed. Most of them were next to self-explanatory one-liners that were usually declarations so comments seemed redundant.

[samee]

I can't really tell which commits you want me to cherry pick on top of float-support branch. I tried commit b790d67: Parity with samee/obliv-c branch float-support, but it still has a lot of diffs with all of my branches here. Can you please list out a few commits that, when played back on float-support, nothing is broken?

E.g.:

git fetch git@github.com:samee/obliv-c.git float-support
git checkout -b samee-float-support FETCH_HEAD

Now you should have two branches, fp-support and samee-float-support. Try to play whatever commits you want on top of my branch, fix things, and push it out for me to see.

Let me know if this works.

[darioncassel]

@samee I apologize for not being clear about that; what I meant is that I have already cherry-picked the last three commits from your float-support branch and integrated them with my branch. I was thinking we could do a merge-rebase of my branch into your repo. Would that work for you? (The reason that my commits won't play back properly on your branch is that my fp-support branch does not have a lot of the commits you included in float-support. I tried to be minimal with what I was including as part of the PR and thus did not include all of those commits. There should not be a difference/problem when it comes to merging though).

[samee]

Merging creates conflicts as well. The conflicts might be simple to solve, though. In any case, if I understand your intentions, you want me to include the following commits from your fp-support branch:

$ git log
commit e13886715bb24638f9d9f42bea766ecca40c2b5c
Author: Darion Cassel <darion.cassel@gmail.com>
Date:   Tue Sep 5 12:11:49 2017 -0400

    Move linreg test to oblivc tests folder, move dbg.h to test/oblivc/common

commit 62acdd3e53e91f0a38e55ead885116a08bbeffde
Author: Darion Cassel <darion.cassel@gmail.com>
Date:   Tue Sep 5 12:03:34 2017 -0400

    Update testFloat with clearer test explanation messages

commit c7d8bed6f1ebe8282cd2998333c2bb81fbc1463f
Author: Darion Cassel <darion.cassel@gmail.com>
Date:   Tue Sep 5 12:00:04 2017 -0400

    Fix compilation-breaking changes in processObliv, move testFloat to oblivc tests folder

commit e39e7886224755302814ed7a773fa4e5ba382ec4
Author: Darion Cassel <darion.cassel@gmail.com>
Date:   Mon Sep 4 17:08:18 2017 -0400

    Add Makefile changes from samee/obliv-c branch float-support

commit a04058390eec5da6300cfd5d3ee670417132faef
Author: Darion Cassel <darion.cassel@gmail.com>
Date:   Mon Sep 4 16:56:47 2017 -0400

    Remove extraneous change markers, improve float op error messages, change genData license

commit b790d6744fc79efffc3114f87890dc6b6210226a
Author: Darion Cassel <darion.cassel@gmail.com>
Date:   Mon Sep 4 16:48:44 2017 -0400

    Parity with samee/obliv-c branch float-support

Is this what I should review now?

[darioncassel]

@samee Actually, of those commits only the top three are needed. The bottom three were just to make my branch functionally equivalent to yours. So these three can be reviewed:

commit e13886715bb24638f9d9f42bea766ecca40c2b5c
Author: Darion Cassel <darion.cassel@gmail.com>
Date:   Tue Sep 5 12:11:49 2017 -0400

    Move linreg test to oblivc tests folder, move dbg.h to test/oblivc/common

commit 62acdd3e53e91f0a38e55ead885116a08bbeffde
Author: Darion Cassel <darion.cassel@gmail.com>
Date:   Tue Sep 5 12:03:34 2017 -0400

    Update testFloat with clearer test explanation messages

commit c7d8bed6f1ebe8282cd2998333c2bb81fbc1463f
Author: Darion Cassel <darion.cassel@gmail.com>
Date:   Tue Sep 5 12:00:04 2017 -0400

    Fix compilation-breaking changes in processObliv, move testFloat to oblivc tests folder

I should note that there were a few changes in this commit:

commit a04058390eec5da6300cfd5d3ee670417132faef
Author: Darion Cassel <darion.cassel@gmail.com>
Date:   Mon Sep 4 16:56:47 2017 -0400

    Remove extraneous change markers, improve float op error messages, change genData license

but as is shown in the commit message, these are fairly straightforward changes based on pervious discussion. You might want to approve the error messages I've added though.

[samee]

Again, Darion, please clean up diffs for commit before mailing them out, either here, or on some other open source project, or even in a company. In particular:

• Make the new series of commits from HEAD so you know they don't conflict. Itis okay if the HEAD moves since you started coding, just rebase them off. Test that merging works before emailing them out.
• Make each commit an easy to understand logically consistent change. If you make them too small, you risk leaving the code in a broken state between commits. If you make them too large, too many unrelated changes show up together, making them harder to understand.

While every project is a little different, here is a reasonable guide to get you started:
https://people.redhat.com/rjones/how-to-supply-code-to-open-source-projects/

For example, I have split up your commit a0405839 into separate smaller ones.

These things affect future maintainability of a project, specially when most of the code is no longer being contributed by a single developer.

When cherry-picking c7d8bed, I saw your comment about removal of | TFloat(k,_) -> fst fnames causing problems. Can you give me an example of this, possibly with the input files? I could be wrong, but I don't see any place where the float-related code is calling that function. setComparisonUS is for the unsigned/signed case, when we need two separate comparison functions based on the signed-ness of operands. setComparison, on the otherhand, is when a single function is enough. Since floats are always signed, we only ever need one function. This is why I removed that line in bc13208cf:

$ git diff bc13208cf{^,} src/ext/processObliv/processObliv.ml
[...]
@@ -501,7 +498,6 @@ let setComparisonUS fnames dest s1 s2 loc =
   let optype = typeOfLval s1 in
   let fname = match unrollType optype with
   | TInt(k,_) -> if isSigned k then snd fnames else fst fnames
-  | TFloat(k,_) -> fst fnames
   | _ -> E.s (E.error "Cannot operate on obliv values of type %a" d_type optype)
   in
   setComparison fname dest s1 s2 loc
[...]
@@ -685,10 +681,10 @@ let rec codegenUncondInstr (instr:instr) : instr = match instr with
         | Shiftlt-> E.s (E.error "Shift operation on float invalid!\n")
         | Ne -> setComparison "__obliv_c__setNotEqualF" v e1 e2 loc
         | Eq -> setComparison "__obliv_c__setEqualToF"  v e1 e2 loc
-        | Lt -> setComparisonUS cmpLtFuncsF v e1 e2 loc
-        | Gt -> setComparisonUS cmpLtFuncsF v e2 e1 loc
-        | Le -> setComparisonUS cmpLeFuncsF v e1 e2 loc
-        | Ge -> setComparisonUS cmpLeFuncsF v e2 e1 loc
+        | Lt -> setComparison "__obliv_c__setLessThanF" v e1 e2 loc
+        | Gt -> setComparison "__obliv_c__setLessThanF" v e2 e1 loc
+        | Le -> setComparison "__obliv_c__setLessThanEqF" v e1 e2 loc
+        | Ge -> setComparison "__obliv_c__setLessThanEqF" v e2 e1 loc

Float-related code shouldn't need to use that function at all. So I'm a little surprised that line caused problems, since I don't think it is being invoked when processing floats. Maybe there's something I missed, but I'd like to know the underlying cause of the problem.

[darioncassel]

@samee Thanks for reviewing my changes. I apologize for the messiness; I did not realize that you wanted to add my changes back to your float-support branch (I was under the impression that we would merge my fp-support branch) and as such I was not looking at the diffs against your branch. For any future changes I will just start at the HEAD of your branch as you suggest so that changes are easier to integrate. Sorry about that.

With regards to the issue with:

-  | TFloat(k,_) -> fst fnames

The reason that I came to think it was an issue was that my floating point test suite, testFloat (now found in test/oblivc/testFloat) was not compiling with that line missing. The error I receive is:

Error: Cannot operate on obliv values of type float obliv __attribute__((__simplifyTemp__,
__implicitCast__))

which of course is the fall-through branch of the match statement that the removed line belongs to. Now that I consider it again though, this could just mean that I have some malformed code in my testFloatOps.oc file, however at the moment I don't see the issue. Let me know what you think.

[samee]

No problem.

Regarding those TFloat cases, yeah let's not try to solve a problem we can't reproduce or understand. We can remove those pieces.

So for the rest of darioncassel@c7d8bed, it seems you added a case for casts to float, but this doesn't seem to be solving everything. E.g., I tried making the following changes on top of darioncassel@e138867:

$ git diff
diff --git a/test/oblivc/testFloat/testFloatOps.oc b/test/oblivc/testFloat/testFloatOps.oc
index 9ec8a407..f09a0efd 100644
--- a/test/oblivc/testFloat/testFloatOps.oc
+++ b/test/oblivc/testFloat/testFloatOps.oc
@@ -37,6 +37,14 @@ void floatOps(void* args)
     printf("Y_r: %f\n", y_r);
     printf("\n");

+    // Casts
+    obliv int i = obliv_x;
+    int i_r;
+    revealOblivInt(&i_r, i, 0);
+    printf("Casting to int test: expecting 1\n");
+    printf("Casting to int: %d\n", i_r);
+    printf("\n");
+
     // Addition
     obliv_z = obliv_x + obliv_y;
     revealOblivFloat(&z, obliv_z, 0);

$ make
../../..//bin/oblivcc -DREMOTE_HOST=localhost -O3 -I . testFloatOps.oc testFloatOps.c ../common/util.c
testFloatOps.oc: In function ‘floatOps’:
testFloatOps.oc:41:3: error: conversion to non-scalar type requested
     obliv int i = obliv_x;
   ^

We want to follow normal C semantics as closely as possible to avoid surprises. In C, we can convert a float to int either implicitly or by explicit casting (whether or not I agree with the implicit casting behavior is a different story). So you may want to add a truncation circuit so this can be done in their obliv counterparts as well, at least if the user wants a conversion.

Or alternatively, you can leave it open as a bug for now, but at least have a clearer error message that says the feature is unimplemented.

[samee]

Btw, I like the way you keep input variables separate in load_data of testFloatOps.c. I think it is less confusing than the way I tend to do it with both parties using the same variable. It may actually help those unfamiliar with Obliv-C.

[darioncassel]

Ok, I agree with what you said. To make sure we're on the same page, going forward are these the things that will be done?

1. Keep the | TFloat(k,_) -> fst fnames case (since it handles casts to floats)
2. Add an error message for the case obliv int <= obliv float
3. Open an issue for the implementation of a truncation circuit for that case.

If you're in agreement, I will go ahead and add the improved error message and then open an issue for the truncation circuit.

[samee]

2. Add an error message for the case obliv int <= obliv float
3. Open an issue for the implementation of a truncation circuit for that case.

Yes, that sounds quite reasonable. Please make sure we cover all combinations of float/int, float/float/ int/float, int/int combinations, even if it just says "unimplemented". We don't want an undefined behavior.

1. Keep the | TFloat(k,_) -> fst fnames case (since it handles casts to floats)

No, please remove it. My understanding if your previous message is that we cannot explain the purpose of those lines, in which case I don't want it in the code.

Let me know if you have more questions.

[darioncassel]

@samee I've been investigating the | TFloat(k,_) -> fst fnames case in setComparisonUS and have found out why it was "necessary" earlier, and the fix for this issue.

The reason why it seemed necessary was actually because of a bug in the logic of the Set(v,UnOp(op,Lval e,t),loc) and Set(v,BinOp(op,Lval e1,Lval e2,t),loc) branches of codegenUncondInstr. Namely, when I was writing the relevant additions necessary to support the obliv float I did not consider the fact that some operators change the expression type t to _Bool obliv and since a bool is just an int, this is actually captured by the TInt(kind,a) when hasOblivAttr a branch. Thus, obliv float comparisons were falling into the setComparisonUS function, and this was actually was not an issue of a cast (not) occurring for the case of obliv float => obliv int as we had thought.

The fix for this that I've implemented in

commit 3ae1c4110ac4a004b6070194822af4acb1b36b6c
Author: Darion Cassel <darion.cassel@gmail.com>
Date:   Tue Sep 19 20:06:00 2017 -0400

    Modify Set(v,UnOp.. and Set(v,BinOp.. to properly handle operator of type obliv float -> obliv bool

is to add an additional check to the TInt(kind,a) when hasOblivAttr a branches to look for cases where the operator(s) are actually obliv floats. (An error of "unimplemented..." will be raised in the case that only one of the operators is of the type of obliv float in the BinOp case.) This approach seems clumsy to me, so if you have an idea for a better way of doing this that'd be great.

A consequence of this fix is that I discovered that the obliv_bits header actually had two errors:

1. Incorrect capitalization of __obliv_c__setLessThanF
2. Missing declaration for __obliv_c__setEqualToF

Those fixes are found in the two commits previous to 3ae1c41.

My intention was to apply this commits on top of your float-support branch but unfortunately, attempting to merge and rebase your branch on top of mine caused a lot of problems so I wasn't able to go through with it... luckily these changes are small and localized enough to be copy & pasted if you'd like or if you'd prefer I can try again rebasing my branch again.

[samee]

Wow, I do appreciate you being thorough, but you are probably working too hard here! I was going through your code in darioncassel@3ae1c41, and you have added lots of code that are unreachable and unnecessary :) Let me explain.

I did not consider the fact that some operators change the expression type t to _Bool obliv and since a bool is just an int, this is actually captured by the TInt(kind,a) when hasOblivAttr a branch.

First of all, great catch! Even I didn't notice it, and I should have. But this conversion is rare. The way CIL works is that it simplifies any C code into a sort of assembly-language-like version of simplified C, before Obliv-C even touches it. Here's a useful experiment you can do. Let us, for instance, try an input mixing ints with floats:

float z = 5.0f + 10;

You can see what CIL does to it using this command (put the line above in a function):

$ path/to/obliv-c/bin/cilly mytest.c -c --save-temps --dosimplify

CIL's simplify plugin will translate it into this (along with other preprocessing cruft we don't care about):

  float z ;
  float __cil_tmp2 ;

  __cil_tmp2 = (float )10;
  z = 5.0f + __cil_tmp2;

And this is normal C code, with no hint of obliv anywhere. The important part here is that, by the time processObliv.ml gets involved, there is no binary expression that deals with mixed operands. The arithmetic addition operator (PlusA) will either have only ints or only floats, never both.

So I don't think line 671 of darioncassel@3ae1c41 will ever be executed. I tried a few tests with your testFloat.oc as well, and I think this is true for most binary ops you have here; you can remove these lines. You'll only need the comparisons, where two floats can produce a bool, since destination type in a comparison differs from operand type. As such, I would also change the pattern-matches a bit to make sure it only matches with destination type bool. So we could change these lines of your commit above (648-652 in processObliv.ml):

    | TInt(kind,a) when hasOblivAttr a ->
        begin match unrollType (typeOf (Lval e1)) with
        | TFloat(kind,a) when hasOblivAttr a ->
          begin match unrollType (typeOf (Lval e2)) with
          | TFloat(kind,a) when hasOblivAttr a ->

into

    | TInt(IBool, a) when hasOblivAttr a &&
                          isOblivFloat (typeOf (Lval e1)) &&
                          isOblivFloat (typeOf (Lval e2))  ->
       (* handle comparisons between floats *)
    | TInt(kind,a) when hasOblivAttr a ->
       (* leave obliv int binary operations unchanged *)

Similarly, bitwise operations on floats: CIL already complains about it long before it gets to your code. You don't need those cases. The situation is similar for unary operators: you have a new BNot case that can be removed. Binary NOT doesn't make sense on float, and earlier stages will already catch that. The only one that CIL won't already catch is logical NOT. You can leave that as is (raising the error) or just implement it with a one-line C function that will call __obliv_c__setNotEqualF and compare with zero.

Let me know if these make sense, or if I missed some case. You can play around with testFloatOps.oc to check my claims. But if you agree, I'll add these changes on.

[darioncassel]

Thanks for going through that Samee! I agree with your analysis of the situation; those cases I added do seem unnecessary given the transformations you showed that CIL performs. I think that the simplification you have listed:

| TInt(IBool, a) when hasOblivAttr a &&
                          isOblivFloat (typeOf (Lval e1)) &&
                          isOblivFloat (typeOf (Lval e2))  ->
       (* handle comparisons between floats *)
| TInt(kind,a) when hasOblivAttr a ->
       (* leave obliv int binary operations unchanged *)

is valid. Please let me know if there is anything else you need me to do.

[jackdoerner]

I found a bug in samee's float-support branch. See line obliv_bits.c, line 2069. This belongs in setupOblivFloat rather than feedOblivFun. This was previously discussed in my code review of Darion's original PR (somewhere above), but somehow got missed by Samee's pass through the repo. I can confirm that moving this line to setupOblivFloat causes libACK's ORAM tests, incl. sqrt, circuit, and floram to pass (on my mac). This test suite is reasonably comprehensive - In addition to initialization tests, each ORAM gets a full pass of sequential writes and reads, and also 2n random reads and writes (where n is the size of the ORAM) of random values. On the other hand, I cannot confirm that floating point ops actually work.

[samee]

Sounds fair. See if this fixes it.

[jackdoerner]

Can confirm that it now works!

[samee]

Merge pushed. Obliv-C now has 'obliv float'.

Thanks both of you for all the effort, specially Darion for implementing the whole thing! Closing out.

[evansuva]

Excellent! Thanks to all of you for your hard work on this. I much appreciate Samee's efforts to preserve quality and consistency in the code, all of Jack's contributions to make this work and go through the PRs, and Darion's work on this and patience in seeing things through.