Skip to content

OctaveDouble precision differences due to octave hex representation  #3

@karlulricsimon

Description

@karlulricsimon

In testIoOctaveDouble.java I added the following test

    @Test
    public void testPrecision() {
    	final OctaveEngine octave = new OctaveEngineFactory().getScriptEngine();
    	octave.eval("format long");
    	octave.eval("a = (1/3) * 0.29");
    	OctaveDouble a = octave.get(OctaveDouble.class, "a");
    	//This assertion will fail.
    	//On evaluation, the console will report "a =  0.0966666666666667",
    	//But retrieving the value back from the wrapper reports the native-hex value cast to a double
    	assertEquals(0.096666666666667, a.get(1), 0.0);
    }

Within the command window of octave, the value of the variable a will be "0.0966666666666667", but if the format is changed to 'native-hex' the value of a is 'be58f28b25bfb83f' which converts to "0.09666666666666665"

It seems that this is a bug in how the wrapper stores the variable value, though it could be that my expectations are incorrect.

I'm using Ubuntu 16.04
octaveInJava version 0.6.8
Octave version 4.3.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions