New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable off heap access for arrays through proxies #21
Comments
You can get array like access using this technique, see below : @test
} package net.openhft.chronicle.map.fromdocs; import net.openhft.lang.model.constraints.MaxSize; public interface BondVOInterface {
}
|
Hmmm what is @maxsize needed for or what is its impact? Is it safe to set max size to some really big number say: 2,147,483,647 (max int) since I only know the real size at runtime. |
Hmm even @maxsize(1000) is not working for doubles ... I am thinking about using java tools to compile a class at runtime with the correct sized annotation. But therefore I would need to know how I can create bigger arrays then 10. Lets say something in the rage 10'000 to 100'000. Do you have an Idea how I could create such big arrays? |
At the moment we simulate arrays through indexed access. public interface IArray { This avoid dealing with arrays directly or the garbage overhead. It not On 23 April 2015 at 16:23, KIC notifications@github.com wrote:
|
Instead of using the generated classes to do this, if you need plain public class DoubleArray implements Byteable {
ArrayIndexOutOfBoundsException();
ArrayIndexOutOfBoundsException();
IllegalArgumentException();
} You can create a DoubleArray by its capacity alone or by doing a get(key) On 23 April 2015 at 17:18, KIC notifications@github.com wrote:
|
Hmm tanks for this class! I really appreciate your help, sadly I can not get this work by myself. I truly have tried to figure this out by myself but this C style memory calculations are to wired for me :-) It will throw a IllegalArgumentException at: public void setData(double[] doubles) { if (doubles.length > capacity()) throw new IllegalArgumentException(); <<<< --- here bytes.writeInt(LENGTH + offset, doubles.length); for (int index = 0; index < doubles.length; index++) bytes.writeDouble(BASE + offset + index * 8L, doubles[index]); } for: public TestArray(double[] data) { this.data = new DoubleArray(data.length); this.data.setData(data); } The problem is that capacity() always returns 0. I am not sure if the constant LENGTH=0 is the problem. |
I was forgetting to set the capacity which is why it was always zero. I The LENGTH, CAPACITY and BASE are the offset of those fields. On 24 April 2015 at 12:05, KIC notifications@github.com wrote:
|
Thanks! I have just found the class but not a unit test, have you pushed the test? https://github.com/OpenHFT/Chronicle-Map/search?utf8=%E2%9C%93&q=DoubleArray This class is slightly different but still does not work. The cpacity() method relies only on two variables the constant CAPACITY and offset: public int capacity() { return bytes.readInt(CAPACITY + offset); } But we never change either CAPACITY nor offset (not in the constructor and not in the setters) public DoubleArray(int capacity) { bytes = DirectStore.allocate(BASE + capacity * 8L).bytes(); offset = 0; } public void setData(double[] doubles) { if (doubles.length > capacity()) throw new IllegalArgumentException(); bytes.writeInt(LENGTH + offset, doubles.length); for (int index = 0; index < doubles.length; index++) bytes.writeDouble(BASE + offset + index * 8L, doubles[index]); } So what we will get is (regadless of our construction parameter): bytes.readInt(4 + 0); |
Correct, I have now uploaded the classes I sent as attachments. I guess https://github.com/OpenHFT/Chronicle-Map/tree/master/src/test/java/net/openhft/lang/values When you do bytes.readInt(offset) you are providing where to find the On 25 April 2015 at 09:15, KIC notifications@github.com wrote:
|
Interestingly one will get a NPE when trying to use the DoubleArray as value in a Map (even as property of another object). Interestingly because this still happens if I provide a default constructor setting some size. I really have no idea how ChronicalMap is calling the method "length" without using a instantiated object. Exception in thread "main" java.lang.IllegalStateException: java.lang.NullPointerException at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.initSize(ByteableMarshaller.java:60) at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller$Default.(ByteableMarshaller.java:195) at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.of(ByteableMarshaller.java:39) at net.openhft.chronicle.map.SerializationBuilder.configureByDefault(SerializationBuilder.java:134) at net.openhft.chronicle.map.SerializationBuilder.(SerializationBuilder.java:97) at net.openhft.chronicle.map.ChronicleMapBuilder.(ChronicleMapBuilder.java:169) at net.openhft.chronicle.map.ChronicleMapBuilder.of(ChronicleMapBuilder.java:191) at com.mindbusters.mql2.test.hft.HftTest.main(HftTest.java:47) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140) Caused by: java.lang.NullPointerException at com.xxx.mql2.test.hft.DoubleArray.length(DoubleArray.java:47) at com.xxx.mql2.test.hft.DoubleArray.maxSize(DoubleArray.java:43) at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.initSize(ByteableMarshaller.java:58) ... 12 more DoubleArray a = new DoubleArray(10); a.setData(new double[]{1, 2, 3, 4, 5}); DoubleArray b = new DoubleArray(10); b.setData(new double[]{5, 6, 7, 8, 9}); ChronicleMap proxyMap = ChronicleMapBuilder.of(Integer.class, DoubleArray.class).create(); proxyMap.put(1, a); proxyMap.put(2, b); System.out.println(proxyMap.get(1)); System.out.println(proxyMap.get(2)); |
I suggest you use an interface like DoubleArray10 example below
} @test
}
|
But this brings us back the fact that I have to know the size at compile time. But I only know the size at runtime. So back to field one and compile interfaces at runtime using java tools and the MaxSize annotation? |
I have added a test to DoubleArrayTest. For now I have added a hack to On 28 April 2015 at 12:35, KIC notifications@github.com wrote:
|
Yes this works great!! Thanks a very lot! System.out.println(proxyMap.getUsing(2, using).getDataAt(2)); |
By using getUsing, you should be able to work garbage free. If you are
|
First of all I really like openHFT, thanks for sharing!!
I try to follow the example using the off heap direct access to primitives ( https://github.com/OpenHFT/Chronicle-Map#off-heap-storage-and-how-using-a-proxy-object-can-improve-performance ). Sadly this seems not to work with arrays. But this would be a very cool thing. Even better would be if we could act on an specific index of an array directly off heap!
See this quick and dirty example:
Hello World
Interface
Implementation
The text was updated successfully, but these errors were encountered: