"LIMIT_EXCEEDED: Too many files in retrieve call, limit is: 10000"

[kaulonline]

Hey Folks,

While retrieving got the Salesforce error - "LIMIT_EXCEEDED: Too many files in retrieve call, limit is: 10000" . This is the hard Salesforce limit which doesn't let you retrieve more than 10000 components in a single API call.

Not sure, if you are already aware of this and have a plan to handle this in the tool .

Pls. suggest.

Thanks,
Ashish

[dieffrei]

@kaulonline it is a salesforce development platform limit.
May you should avoid do retrieve some metadata types like documents/reports or retrieve in 2 steps

[kaulonline]

Thanks Diéffrei. Agree, this is a salesforce limit and this will be encountered during any metadata calls > 10000 components , however wanted to know if there was any additional mechanism built to either loop through or split the retrieve request.

[dieffrei]

Got it!
I think there's no feature built-in on force-dev-tool to help you with this, because I had the same problem 1 year a go and received the same answer.
BWT I think it will be very nice if force-dev-tool could managed more than 10K of components.

[kaulonline]

Yeah.. It would be interesting to see this tool handle > 10K component. Meanwhile, I tried to split the package and keeping a limited types and got the below error :

``root@ubuntu:/opt# force-dev-tool retrieve
Retrieving from remote mydev1 to directory src


<--- Last few GCs --->

[10601:0x3b32a50]  1052490 ms: Mark-sweep 111.2 (145.3) -> 110.7 (145.8) MB, 17.4 / 0.0 ms  (+ 6.7 ms in 5 steps since start of marking, biggest step 3.3 ms, walltime since start of marking 621 ms) (average mu = 0.999, current mu = 0.996) allocation failu[10601:0x3b32a50]  1059471 ms: Mark-sweep 390.7 (425.9) -> 381.9 (420.9) MB, 153.4 / 0.0 ms  (+ 32.5 ms in 1 steps since start of marking, biggest step 32.5 ms, walltime since start of marking 6004 ms) (average mu = 0.997, current mu = 0.978) allocation f

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x188e32b041bd]
    1: StubFrame [pc: 0x188e32e2c2a0]
Security context: 0x35ed0bd9e6c9 <JSObject>
    2: lookupValue(aka lookupValue) [0x3596b966e7d1] [/usr/local/lib/node_modules/force-dev-tool/node_modules/jsforce/lib/soap.js:135] [bytecode=0x64014256d1 offset=26](this=0x1ec229c022e1 <undefined>,obj=0xfdc37d82201 <Very long string[48565345]>,propRegExps=0x1e6da9082259 <JSArray[2]>)
    3: getResponseBody [0x28a13c4...

FATAL ERROR: invalid table size Allocation failed - JavaScript heap out of memory
 1: 0x89c2f0 node::Abort() [node]
 2: 0x89c33c  [node]
 3: 0xa8f05e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xa8f278 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xe7cad2  [node]
 6: 0xf8d30a v8::internal::OrderedHashTable<v8::internal::OrderedHashSet, 1>::Allocate(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [node]
 7: 0xfb1475 v8::internal::OrderedHashTable<v8::internal::OrderedHashSet, 1>::Rehash(v8::internal::Handle<v8::internal::OrderedHashSet>, int) [node]
 8: 0xfb17b9 v8::internal::OrderedHashTable<v8::internal::OrderedHashSet, 1>::EnsureGrowable(v8::internal::Handle<v8::internal::OrderedHashSet>) [node]
 9: 0xfb18b0 v8::internal::OrderedHashSet::Add(v8::internal::Handle<v8::internal::OrderedHashSet>, v8::internal::Handle<v8::internal::Object>) [node]
10: 0xf4ea75 v8::internal::KeyAccumulator::AddKey(v8::internal::Handle<v8::internal::Object>, v8::internal::AddKeyConversion) [node]
11: 0xdff042  [node]
12: 0xf5097b v8::internal::KeyAccumulator::CollectOwnElementIndices(v8::internal::Handle<v8::internal::JSReceiver>, v8::internal::Handle<v8::internal::JSObject>) [node]
13: 0xf51e22 v8::internal::KeyAccumulator::CollectOwnKeys(v8::internal::Handle<v8::internal::JSReceiver>, v8::internal::Handle<v8::internal::JSObject>) [node]
14: 0xf52b15 v8::internal::KeyAccumulator::CollectKeys(v8::internal::Handle<v8::internal::JSReceiver>, v8::internal::Handle<v8::internal::JSReceiver>) [node]
15: 0xf52f49 v8::internal::FastKeyAccumulator::GetKeys(v8::internal::GetKeysConversion) [node]
16: 0x10d98a3 v8::internal::Runtime_ForInEnumerate(int, v8::internal::Object**, v8::internal::Isolate*) [node]
17: 0x188e32b041bd
Aborted (core dumped)
``

[dieffrei]

hmmmm yeap I had the same problem too. (#118)
Seems you organization is big!
Did you try to remove Documents or some Irrelevant metadata?

[kaulonline]

Thanks for pointing to the same issue. Yeah this seems to be the similar issue. Although, I had taken out Reports/Dashboards/Groups/Roles and Document (+many more unnecessary ones)

[amtrack]

There are no plans for force-dev-tool to implement something extending the Salesforce limit LIMIT_EXCEEDED: Too many files in retrieve call, limit is: 10000.

However you can try working around this limit with a simple wrapper bash script.

The scripts splits the given patterns from the package.xml into an extra package.xml and retrieves the two directories separately. After retrieving, it merges back the two directories (including the package.xml files).

The patterns must only match Metadata which is safe to be retrieved without any CustomObject, such as Document, Role, PermissionSet etc. (NOT Profile, CustomApplication, ...).

Example

$ force-dev-tool fetch myremote
$ force-dev-tool package -a myremote
$ RETRIEVE_SPLIT_PATTERNS='PermissionSet/*,Document/*' bash force-dev-tool-retrieve-wrapper.sh myremote