Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Joining and sorting ranks fails to load serialized double array holding page rank scores #5

Closed
sebastian-nagel opened this issue Sep 5, 2022 · 3 comments
Closed

Joining and sorting ranks fails to load serialized double array holding page rank scores #5

sebastian-nagel opened this issue Sep 5, 2022 · 3 comments
Labels
bug
Milestone
0.1

Comments

@sebastian-nagel
Copy link
Contributor 

[main] INFO org.commoncrawl.webgraph.JoinSortRanks - Loading page rank values from host/cc-main-2021-22-oct-nov-jan-host-pagerank.ranks
Exception in thread "main" java.lang.IllegalArgumentException: newLimit < 0: (-1216172000 < 0)
        at java.base/java.nio.Buffer.createLimitException(Buffer.java:372)
        at java.base/java.nio.Buffer.limit(Buffer.java:346)
        at java.base/java.nio.ByteBuffer.limit(ByteBuffer.java:1107)
        at java.base/java.nio.MappedByteBuffer.limit(MappedByteBuffer.java:235)
        at java.base/java.nio.MappedByteBuffer.limit(MappedByteBuffer.java:67)
        at it.unimi.dsi.fastutil.io.BinIO.loadDoubles(BinIO.java:6431)
        at it.unimi.dsi.fastutil.io.BinIO.loadDoubles(BinIO.java:6452)
        at it.unimi.dsi.fastutil.io.BinIO.loadDoubles(BinIO.java:6520)
        at it.unimi.dsi.fastutil.io.BinIO.loadDoubles(BinIO.java:7006)
        at it.unimi.dsi.fastutil.io.BinIO.loadDoubles(BinIO.java:7018)
        at org.commoncrawl.webgraph.JoinSortRanks.loadPageRank(JoinSortRanks.java:50)
        at org.commoncrawl.webgraph.JoinSortRanks.main(JoinSortRanks.java:319)
  • worked around (no time at this point for a deeper analysis) by downgrading (reverting recent commits to f33704b used to build the preceding graph)
sebastian-nagel added a commit that referenced this issue Sep 5, 2022
@sebastian-nagel
Add unit test to reproduce #5 (loading of a serialized double array
49174af 
in a 2 GiB file fails)
@sebastian-nagel
Copy link
Contributor Author

sebastian-nagel commented Sep 5, 2022 •
edited
Loading

The failure loading of a double array is reproducible with any array if the size of the serialization file is 2 GiB or more, see 49174af. Downgrading fastutil-core to 8.5.7 fixes the issue, the unit test also passes using the current head of fastutil (vigna/fastutil@b813824). See also question about release of fastutil 8.5.9.

sebastian-nagel added a commit that referenced this issue Sep 13, 2022
@sebastian-nagel
Add unit test to reproduce #5 (loading of a serialized double array
7131710 
in a 2 GiB file fails)
@sebastian-nagel
Copy link
Contributor Author

Fixed by downgrading to fastutil-core 8.5.7

@sebastian-nagel sebastian-nagel added this to the 0.1 milestone Sep 15, 2022
sebastian-nagel added a commit that referenced this issue Sep 18, 2022
@sebastian-nagel
HostToDomainGraph: allow public suffixes as domain names (fixes #1)
d44e946 
- rename existing option `--no-strict-domain-validate` into
  `--multipart-suffixes-as-domains`, complete documentatin
- add unit tests

Log progress when processing large graphs

HostToDomainGraph / JoinSortRanks: input/output is always UTF-8
(fixes #5): write output always as UTF-8
sebastian-nagel added a commit that referenced this issue Sep 20, 2022
@sebastian-nagel
HostToDomainGraph: allow public suffixes as domain names (fixes #1)
a223d1e 
- rename existing option `--no-strict-domain-validate` into
  `--multipart-suffixes-as-domains`, complete documentatin
- add unit tests

Log progress when processing large graphs

HostToDomainGraph / JoinSortRanks: input/output is always UTF-8
(fixes #5): write output always as UTF-8
sebastian-nagel added a commit that referenced this issue Sep 20, 2022
@sebastian-nagel
Unit test to reproduce #5: configure Maven to use sufficient heap
cac8936 
space to execute unit test using large arrays
sebastian-nagel added a commit that referenced this issue Sep 20, 2022
@sebastian-nagel
Unit test to reproduce #5: configure Maven to use sufficient heap
a2658f3 
space to execute unit test using large arrays
sebastian-nagel added a commit that referenced this issue Sep 20, 2022
@sebastian-nagel
HostToDomainGraph: allow public suffixes as domain names (fixes #1)
7be075b 
- rename existing option `--no-strict-domain-validate` into
  `--multipart-suffixes-as-domains`, complete documentatin
- add unit tests

Log progress when processing large graphs

HostToDomainGraph / JoinSortRanks: input/output is always UTF-8
(fixes #5): write output always as UTF-8
@vigna
Copy link

vigna commented Sep 26, 2022

Thank you for reporting this. Fixed in fastutil 8.5.9.

sebastian-nagel added a commit that referenced this issue Mar 15, 2023
@sebastian-nagel
Reduce Java heap memory occupied by unit test to reproduce #5
4376811 
(loading of a serialized double array in a 2 GiB file fails)
- reset array after storing it and call garbage collector to free memory
- more verbose logging after steps in unit test
@sebastian-nagel sebastian-nagel mentioned this issue Apr 9, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
0.1
Development

No branches or pull requests
2 participants
@vigna @sebastian-nagel