Workaround/bids validator

[jdkent]

Hi,

Another member of my lab was interested in using this utility, but we kept on getting stuck with the bids-validator getting an error like this

<--- Last few GCs --->

  426429 ms: Scavenge 1402.9 (1457.3) -> 1402.9 (1457.3) MB, 3.8 / 0 ms (+ 1.3 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
  427370 ms: Mark-sweep 1402.9 (1457.3) -> 1401.6 (1457.3) MB, 940.5 / 0 ms (+ 1.3 ms in 1 steps since start of marking, biggest step 1.3 ms) [last resort gc].
  428305 ms: Mark-sweep 1401.6 (1457.3) -> 1401.6 (1457.3) MB, 935.2 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2990767b4629 <JS Object>
    2: getFiles(aka getFiles) [/usr/local/lib/node_modules/bids-validator/utils/files.js:~97] [pc=0x178a22475a7e] (this=0x2990767041b9 <undefined>,dir=0x1898105ee731 <String[107]: /home/jdkent/mnt/vosslabhpc/Projects/Bike_ATrain/Imaging/BIDS/sourcedata/RAW_BIDS/sub-GEP385/ses-passivepre>,files_=0x1226ebeb1379 <JS Array[4918904]>)
    3: getFiles(aka getFiles) [/usr/local/lib/node_modules/bids-v...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
Aborted (core dumped)

I played with the --config option for bids-validator with no avail, and for the interest of time, I provided this workaround to make the container run. I think the error is related to bids-validator trying to read into sourcedata and derivatives, but I couldn't get the --config option to work for me.

P.S.
I also did some flake8 conforming with the code so that is more in line with PEP 8 standards.

best,
James

[chrisgorgo]

Is this happening with most recent version of the validator? Have you tried updating node and/or using this branch https://github.com/INCF/bids-validator/pull/346?

[jdkent]

sorry, I should have included versions, it's happening with the versions installed in the docker container, and with version bids-validator: 0.23.11; node: v4.2.6 when I try it as it's installed on my computer (Ubuntu 16.04). I have not tried with branch INCF/bids-validator#346, I will try that next.

EDIT: realized from the comments in INCF/bids-validator#346, that I didn't exclude dot files and dot directories, which may be the problem since I'm using datalad with this dataset.

[chrisgorgo]

Might be also worth updating node.js

[jdkent]

I repeated the process with node: v6.11.5, but got a similar result:

bids-validator --config bids-validator.config ~/mnt/vosslabhpc/Projects/Bike_ATrain/Imaging/BIDS

<--- Last few GCs --->

  384744 ms: Mark-sweep 1377.8 (1437.8) -> 1377.8 (1437.8) MB, 1287.2 / 0.0 ms [allocation failure] [GC in old space requested].
  385984 ms: Mark-sweep 1377.8 (1437.8) -> 1377.8 (1437.8) MB, 1240.0 / 0.0 ms [allocation failure] [GC in old space requested].
  387265 ms: Mark-sweep 1377.8 (1437.8) -> 1377.8 (1417.8) MB, 1280.3 / 0.0 ms [last resort gc].
  388510 ms: Mark-sweep 1377.8 (1417.8) -> 1377.8 (1417.8) MB, 1245.1 / 0.0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2623298cf781 <JS Object>
    2: getFiles [/usr/local/lib/node_modules/bids-validator/utils/files.js:102] [pc=0x2673977873f] (this=0x8a8b9d8db29 <JS Global Object>,dir=0x35537f496249 <String[168]: /home/jdkent/mnt/vosslabhpc/Projects/Bike_ATrain/Imaging/BIDS/sourcedata/RAW_BIDS/sub-GEP385/ses-activepost/func/sub-GEP385_ses-activepost_task-spatialnback_run-01_bold>,files_=0x1b7c942e5369 <JS Array[4861460]>)
    3: getFil...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [bids-validator]
 2: 0x109f13c [bids-validator]
 3: v8::Utils::ReportApiFailure(char const*, char const*) [bids-validator]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [bids-validator]
 5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [bids-validator]
 6: v8::internal::Factory::NewScopeInfo(int) [bids-validator]
 7: v8::internal::ScopeInfo::Create(v8::internal::Isolate*, v8::internal::Zone*, v8::internal::Scope*) [bids-validator]
 8: 0xb4f7b3 [bids-validator]
 9: 0xb56e46 [bids-validator]
10: v8::internal::Compiler::Compile(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Compiler::ClearExceptionFlag) [bids-validator]
11: v8::internal::Runtime_CompileLazy(int, v8::internal::Object**, v8::internal::Isolate*) [bids-validator]
12: 0x267394092a7
Aborted (core dumped)

bids-validator still appears to be traversing the sourcedata directory despite me trying to tell it not to.
I've tried this with the config file either containing this:

{
	"ignoredFiles": ["derivatives/**", "sourcedata/**"]
}

or this:

{
	"ignoredFiles": ["/derivatives/**", "/sourcedata/**"]
}

So then my next step should be using the new branch? or am I not specifying the config file correctly?

Should we move this conversation to the bids-validator github?

[chrisgorgo]

That branch should help with avoiding traversing unwanted files. Please give it a try. Best, Chris

…

On Wed, Nov 1, 2017 at 10:15 PM, James Kent ***@***.***> wrote: I repeated the process with node: v6.11.5, but got a similar result: bids-validator --config bids-validator.config ~/mnt/vosslabhpc/Projects/Bike_ATrain/Imaging/BIDS <--- Last few GCs ---> 384744 ms: Mark-sweep 1377.8 (1437.8) -> 1377.8 (1437.8) MB, 1287.2 / 0.0 ms [allocation failure] [GC in old space requested]. 385984 ms: Mark-sweep 1377.8 (1437.8) -> 1377.8 (1437.8) MB, 1240.0 / 0.0 ms [allocation failure] [GC in old space requested]. 387265 ms: Mark-sweep 1377.8 (1437.8) -> 1377.8 (1417.8) MB, 1280.3 / 0.0 ms [last resort gc]. 388510 ms: Mark-sweep 1377.8 (1417.8) -> 1377.8 (1417.8) MB, 1245.1 / 0.0 ms [last resort gc]. <--- JS stacktrace ---> ==== JS stack trace ========================================= Security context: 0x2623298cf781 <JS Object> 2: getFiles [/usr/local/lib/node_modules/bids-validator/utils/files.js:102] [pc=0x2673977873f] (this=0x8a8b9d8db29 <JS Global Object>,dir=0x35537f496249 <String[168]: /home/jdkent/mnt/vosslabhpc/Projects/Bike_ATrain/Imaging/BIDS/sourcedata/RAW_BIDS/sub-GEP385/ses-activepost/func/sub-GEP385_ses-activepost_task-spatialnback_run-01_bold>,files_=0x1b7c942e5369 <JS Array[4861460]>) 3: getFil... FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory 1: node::Abort() [bids-validator] 2: 0x109f13c [bids-validator] 3: v8::Utils::ReportApiFailure(char const*, char const*) [bids-validator] 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [bids-validator] 5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [bids-validator] 6: v8::internal::Factory::NewScopeInfo(int) [bids-validator] 7: v8::internal::ScopeInfo::Create(v8::internal::Isolate*, v8::internal::Zone*, v8::internal::Scope*) [bids-validator] 8: 0xb4f7b3 [bids-validator] 9: 0xb56e46 [bids-validator] 10: v8::internal::Compiler::Compile(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Compiler::ClearExceptionFlag) [bids-validator] 11: v8::internal::Runtime_CompileLazy(int, v8::internal::Object**, v8::internal::Isolate*) [bids-validator] 12: 0x267394092a7 Aborted (core dumped) bids-validator still appears to be traversing the sourcedata directory despite me trying to tell it not to. I've tried this with the config file either containing this: { "ignoredFiles": ["derivatives/**", "sourcedata/**"] } or this: { "ignoredFiles": ["/derivatives/**", "/sourcedata/**"] } So then my next step should be using the new branch? or am I not specifying the config file correctly? Should we move this conversation to the bids-validator github? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#37 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAOkp_NzvHbax4G6h054BoiI279qtjN4ks5syVAOgaJpZM4QO3W2> .

[jdkent]

installed the branch like so:
npm install -g https://github.com/suyashdb/bids-validator.git#optimizeTraversalOfDirectory
and that version worked perfectly, thanks!
Should we still include this pull request to allow users the option to not do a bids-check (if they have already done this themselves) of if they want to use a specific configuration of bids-validator?

[chrisgorgo]

Thanks!