Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

If the order of keys is inconsistent during deserialization, OOM will occur. #9

Closed
linking12 opened this issue Dec 12, 2023 · 6 comments

Comments

@linking12
Copy link

linking12 commented Dec 12, 2023

first deserialization, if key order is:
key1
key2
key3

sencond deserialization, if key order is:
key2
key3
key1

when do deserialize multiple times use compactmap , will trigger OOM

@vlsi
Copy link
Owner

vlsi commented Dec 12, 2023

Do you mean serialize-deserialize within the same JVM or across JVMs?

@linking12
Copy link
Author

@vlsi in my case , it is across JVMs;
but i write a simple code to test this scenes, it will reappear

@linking12
Copy link
Author

linking12 commented Dec 13, 2023

oom simple code(keys are out of order):

public class CompactRandomOrder {
    private static final int ENTRY_CARD = 1000;
    private static final int LOOP_COUNT = 100000;
    private static final int MAP_SIZE = 50;
    private static Random random = new Random(ENTRY_CARD);
    public static void main(String[] args) throws InterruptedException {
        List<String> keyList = new ArrayList<>();
        List<String> valueList = new ArrayList<>();
        for (int i = 0; i < ENTRY_CARD; i++) {
            keyList.add(i + "#keyKeykeyKeykeyKeykeyKeykeyKeykeyKeykeyKeykeyKeykeyKey" + random.nextInt());
            valueList.add(i + "#valueValuevalueValuevalueValuevalueValuevalueValue" + random.nextInt());
        }
        int idx;
        List<Map> listMap = new ArrayList<>(LOOP_COUNT);
        for (int i = 0; i < LOOP_COUNT; i++) {
            CompactHashMap map = new CompactHashMap();
            for (int j = 0; j < MAP_SIZE; j++) {
                idx = Math.abs(random.nextInt()) % ENTRY_CARD;
                map.put(keyList.get(idx), valueList.get(idx));
            }
            listMap.add(map);
        }

        System.out.println("has filled");
        Thread.sleep(200000);
    }
}

normal code (keys are sequential)

public class CompactOrderly {
    private static final int ENTRY_CARD = 1000;
    private static final int LOOP_COUNT = 100000;
    private static final int MAP_SIZE = 50;
    private static Random random = new Random(ENTRY_CARD);
    @SneakyThrows
    public static void main(String[] args)  {
        List<String> keyList = new ArrayList<>();
        List<String> valueList = new ArrayList<>();
        for (int i = 0; i < ENTRY_CARD; i++) {
            keyList.add(i + "#keyKeykeyKeykeyKeykeyKeykeyKeykeyKeykeyKeykeyKeykeyKey" + random.nextInt());
            valueList.add(i + "#valueValuevalueValuevalueValuevalueValuevalueValue" + random.nextInt());
        }
        int idx;
        List<Map> listMap = new ArrayList<>(LOOP_COUNT);
        for (int i = 0; i < LOOP_COUNT; i++) {
            CompactHashMap map = new CompactHashMap();
            idx = Math.abs(random.nextInt()) % (ENTRY_CARD - MAP_SIZE);
            for (int j = idx; j < ENTRY_CARD; j++) {
                if (j - idx >= 50) {
                    break;
                }
                map.put(keyList.get(j), valueList.get(j));
            }
            listMap.add(map);
        }


        System.out.println("has filled");
        Thread.sleep(200000);
    }
}


image

@linking12
Copy link
Author

@vlsi any idea?

@vlsi vlsi closed this as completed in f1b2929 Dec 13, 2023
@vlsi
Copy link
Owner

vlsi commented Dec 13, 2023

I've published 2.0.1, and it should appear on Central soon.

@vlsi
Copy link
Owner

vlsi commented Dec 13, 2023

Thanks for figuring out the root cause.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants