Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RangeError [ERR_OUT_OF_RANGE]: The value of "offset" is out of range #1

Closed
staltz opened this issue Jul 20, 2020 · 7 comments
Closed

RangeError [ERR_OUT_OF_RANGE]: The value of "offset" is out of range #1

staltz opened this issue Jul 20, 2020 · 7 comments

Comments

@staltz
Copy link
Member

staltz commented Jul 20, 2020

First I ran node copy-json-to-bipf.js path/to/log.offset path/to/jitdb.log (which by the way I had to do some fixes to the code, PR incoming soon), then...

Upon running node example path/to/jitdb.log:

loaded indexes []
missing indexes: [ { seek: [Function: seekType],
    value: <Buffer 70 6f 73 74>,
    indexType: 'type',
    indexName: 'type_post' },
  { seek: [Function: seekAuthor],
    value:
     <Buffer 40 51 6c 43 54 70 76 59 37 70 39 74 79 32 79 4f 46 72 76 31 57 55 31 41 45 38 38 61 6f 51 63 34 59 37 77 59 61 6c 37 50 46 63 2b 77 3d 2e 65 64 32 35 ... >,
    indexType: 'author',
    indexName:
     'author_@QlCTpvY7p9ty2yOFrv1WU1AE88aoQc4Y7wYal7PFc+w=.ed25519' } ]
time: 3563ms, total items: 1112291
writing index to indexes/offset.index
writing index to indexes/timestamp.index
saving index:type_post
writing index to indexes/type_post.index
saving index:author_@QlCTpvY7p9ty2yOFrv1WU1AE88aoQc4Y7wYal7PFc+w=.ed25519
writing index to indexes/author_@QlCTpvY7p9ty2yOFrv1WU1AE88aoQc4Y7wYal7PFc+w=.ed25519.index
internal/buffer.js:43
    throw new ERR_OUT_OF_RANGE(type || 'offset', 'an integer', value);
    ^

RangeError [ERR_OUT_OF_RANGE]: The value of "offset" is out of range. It must be an integer. Received NaN
    at boundsError (internal/buffer.js:43:11)
    at Buffer.readUInt16LE (internal/buffer.js:128:5)
    at /home/staltz/oss/manyverse/jitdb/node_modules/.pnpm/flumelog-aligned-offset@1.1.2/node_modules/flumelog-aligned-offset/index.js:141:27
    at Request._callback (/home/staltz/oss/manyverse/jitdb/node_modules/.pnpm/flumelog-aligned-offset@1.1.2/node_modules/flumelog-aligned-offset/index.js:76:7)
    at Request.callback (/home/staltz/oss/manyverse/jitdb/node_modules/.pnpm/random-access-storage@1.4.1/node_modules/random-access-storage/index.js:173:8)
    at onread (/home/staltz/oss/manyverse/jitdb/node_modules/.pnpm/random-access-file@2.1.4/node_modules/random-access-file/index.js:81:31)
    at FSReqWrap.wrapper [as oncomplete] (fs.js:462:17)

Dependencies:

dependencies:
atomic-file 2.0.1
bipf 1.3.0
flumelog-aligned-offset 1.1.2
idb-kv-store 4.4.0
lodash.debounce 4.0.8
mkdirp 1.0.4
push-stream 11.0.0
sanitize-filename 1.6.3
typedfastbitset 0.1.5

devDependencies:
flumecodec 0.0.1
flumelog-offset 3.4.4
obv 0.0.1
pull-stream 3.6.14
rimraf 3.0.2
ssb-keys 7.2.2
ssb-validate 4.1.1
tape 5.0.1

Versions:

{ jitdb: '0.4.3',
  npm: '6.4.1',
  ares: '1.15.0',
  cldr: '33.1',
  http_parser: '2.8.0',
  icu: '62.1',
  modules: '64',
  napi: '3',
  nghttp2: '1.34.0',
  node: '10.15.3',
  openssl: '1.1.0j',
  tz: '2018e',
  unicode: '11.0',
  uv: '1.23.2',
  v8: '6.8.275.32-node.51',
  zlib: '1.2.11' }

My old and new log files:

-rw-r--r--  1 staltz staltz  777M july 20 14:09 jitdb.log
-rw-r--r--  1 staltz staltz  909M july 19 23:43 log.offset

My indexes folder:

-rw-r--r-- 1 staltz staltz 221K july 20 14:14 'author_@QlCT(...)7PFc+w=.ed25519.index'
-rw-r--r-- 1 staltz staltz 3,9M july 20 14:14  offset.index
-rw-r--r-- 1 staltz staltz 7,7M july 20 14:14  timestamp.index
-rw-r--r-- 1 staltz staltz 160K july 20 14:14  type_post.index
@arj03
Copy link
Member

arj03 commented Jul 20, 2020

Woops, that's this FIXME. I'll better get that fixed then ;)

@arj03 arj03 closed this as completed in 9c7a289 Jul 20, 2020
@arj03
Copy link
Member

arj03 commented Jul 20, 2020

I fixed this now, together with some nasty bugs in flumelog-aligned-offset.

@staltz
Copy link
Member Author

staltz commented Jul 21, 2020

@arj03 Very much appreciated that you worked on it quickly!

However, I think I have the same issue still.

$ node example path/to/jitdb.log

loaded indexes [ 'author_@QlCTpvY7p9ty2yOFrv1WU1AE88aoQc4Y7wYal7PFc+w=.ed25519',
  'offset',
  'timestamp',
  'type_post' ]
internal/buffer.js:43
    throw new ERR_OUT_OF_RANGE(type || 'offset', 'an integer', value);
    ^

RangeError [ERR_OUT_OF_RANGE]: The value of "offset" is out of range. It must be an integer. Received NaN
    at boundsError (internal/buffer.js:43:11)
    at Buffer.readUInt16LE (internal/buffer.js:128:5)
    at /home/staltz/oss/manyverse/jitdb/node_modules/.pnpm/flumelog-aligned-offset@1.1.3/node_modules/flumelog-aligned-offset/index.js:148:27
    at Request._callback (/home/staltz/oss/manyverse/jitdb/node_modules/.pnpm/flumelog-aligned-offset@1.1.3/node_modules/flumelog-aligned-offset/index.js:76:7)
    at Request.callback (/home/staltz/oss/manyverse/jitdb/node_modules/.pnpm/random-access-storage@1.4.1/node_modules/random-access-storage/index.js:173:8)
    at onread (/home/staltz/oss/manyverse/jitdb/node_modules/.pnpm/random-access-file@2.1.4/node_modules/random-access-file/index.js:81:31)
    at FSReqWrap.wrapper [as oncomplete] (fs.js:462:17)

Dependencies:

dependencies:
atomic-file 2.0.1
bipf 1.3.0
flumelog-aligned-offset 1.1.3
idb-kv-store 4.4.0
lodash.debounce 4.0.8
mkdirp 1.0.4
push-stream 11.0.0
sanitize-filename 1.6.3
typedfastbitset 0.1.5

devDependencies:
flumecodec 0.0.1
flumelog-offset 3.4.4
obv 0.0.1
pull-stream 3.6.14
rimraf 3.0.2
ssb-keys 7.2.2
ssb-validate 4.1.1
tape 5.0.1

Versions:

{ jitdb: '0.5.1',
  npm: '6.4.1',
  ares: '1.15.0',
  cldr: '33.1',
  http_parser: '2.8.0',
  icu: '62.1',
  modules: '64',
  napi: '3',
  nghttp2: '1.34.0',
  node: '10.15.3',
  openssl: '1.1.0j',
  tz: '2018e',
  unicode: '11.0',
  uv: '1.23.2',
  v8: '6.8.275.32-node.51',
  zlib: '1.2.11' }

Next, I'll try deleting everything and starting over

@staltz
Copy link
Member Author

staltz commented Jul 21, 2020

Okay, deleting and starting over helped. I got a different error, but just a missing dependency bug. All good, you can ignore this issue

@staltz
Copy link
Member Author

staltz commented Jul 21, 2020

It worked!

loaded indexes []
missing indexes: [ { seek: [Function: seekType],
    value: <Buffer 70 6f 73 74>,
    indexType: 'type',
    indexName: 'type_post' },
  { seek: [Function: seekAuthor],
    value:
     <Buffer 40 51 6c 43 54 70 76 59 37 70 39 74 79 32 79 4f 46 72 76 31 57 55 31 41 45 38 38 61 6f 51 63 34 59 37 77 59 61 6c 37 50 46 63 2b 77 3d 2e 65 64 32 35 ... >,
    indexType: 'author',
    indexName:
     'author_@QlCTpvY7p9ty2yOFrv1WU1AE88aoQc4Y7wYal7PFc+w=.ed25519' } ]
growing index
growing index
growing index
growing index
growing index
growing index
growing index
growing index
growing index
growing index
growing index
growing index
growing index
growing index
time: 2658ms, total items: 1112794
writing index to indexes/offset.index
writing index to indexes/timestamp.index
saving index:type_post
writing index to indexes/type_post.index
saving index:author_@QlCTpvY7p9ty2yOFrv1WU1AE88aoQc4Y7wYal7PFc+w=.ed25519
writing index to indexes/author_@QlCTpvY7p9ty2yOFrv1WU1AE88aoQc4Y7wYal7PFc+w=.ed25519.index
get all: 99ms, total items: 2994
get all posts from user: 2782.643ms
results 2994
get values and sort top 10: 7.196ms
get last 10 posts from user: 9.918ms
results 231909
get values and sort top 50: 265.956ms
get top 50 posts: 267.183ms
missing indexes: [ { seek: [Function: seekType],
    value: <Buffer 63 6f 6e 74 61 63 74>,
    indexType: 'type',
    indexName: 'type_contact' } ]
time: 1345ms, total items: 1112794
saving index:type_contact
writing index to indexes/type_contact.index
get all: 1340ms, total items: 217405
contacts: 3210.456ms

@arj03
Copy link
Member

arj03 commented Jul 21, 2020

Excellent :)

@staltz
Copy link
Member Author

staltz commented Jul 21, 2020

Next up, I want to try running this on mobile, to see if ARM architectures (and Android's weird FS) have any impact on performance.

@arj03 arj03 mentioned this issue Jan 13, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@staltz @arj03