Dumping tse files from very big active directory is failing

[vletoux]

Hi,

Thanks for the tool which looks very promissing.

I'm trying to export data from a very large resource active directory (> 100 000 users).
The AD is not near my area so the connection is slow.
I've tried 3 different exports which is failing everytime for the same reason: the object has been deleted since the export started.
The failure occurs in ldapdump.c line 909.

I've tried as an alternative a ntds.dit export but dsamain seems unable to mount it (with a message about the corruption of second level indexes). Same problem when trying to mount it with an AD LDS.

Is it possible to fix the case where the object has been deleted since the beginning of the export ?
I would have it done my self but there is a lot of impacts in the calling functions.

* Command: .\Bin\LdapDump.exe -x 'INFO' -f 'C:\Users\alpha\Documents\AD-control-paths-master\Dump\\logs\tatayoyo.ldpdmp.log' -a 'C:\Users\alpha\Documents\AD-control-paths-master\Dump\\dumps\tatayoyo.ace.ldpdmp.tsv' -o 'C:\Users\alpha\Documents\AD-control-paths-master\Dump\\dumps\tatayoyo.obj.ldpdmp.tsv' -c 'C:\Users\alpha\Documents\AD-contr ol-paths-master\Dump\\dumps\tatayoyo.sch.ldpdmp.tsv' -s '10.10.10.10'
[07:59:31] [+] Start
[07:59:31] [.] -- Setting log level to <INFO>
[07:59:31] [.] -- Setting log file to <C:\Users\alpha\Documents\AD-control-paths-master\Dump\\logs\tatayoyo.ldpdmp.log>
[07:59:31] [.] -- LDAP server <10.10.10.10:389>
[07:59:31] [.] -- LDAP implicit authentication with username <alpha>
[07:59:31] [.] -- Dumping <ace:C:\Users\alpha\Documents\AD-control-paths-master\Dump\\dumps\tatayoyo.ace.ldpdmp.tsv> <obj:C:\Users\alpha\Documents\AD-control-paths-master\Dump\\dumps\tatayoyo.obj.ldpdmp.tsv> <sch:C:\Users\alpha\Documents\AD-control-paths-master\Dump\\dumps\tatayoyo.sch.ldpdmp.tsv>
[07:59:31] [.] -- Dumping data on default NC (domain/config/schema)
[07:59:31] [+] Connecting to LDAP server...
[07:59:31] [.] -- Domain NC: <DC=tatayoyo,DC=LOCAL>
[07:59:31] [.] -- Config NC: <CN=Configuration,DC=tatayoyo,DC=LOCAL>
[07:59:31] [.] -- Schema NC: <CN=Schema,CN=Configuration,DC=tatayoyo,DC=LOCAL>
[07:59:31] [+] Dumping ACEs into <C:\Users\alpha\Documents\AD-control-paths-master\Dump\\dumps\tatayoyo.ace.ldpdmp.tsv>
[09:11:30] [.] -- Count: <288278>
[09:11:30] [+] Dumping Object list into <C:\Users\alpha\Documents\AD-control-paths-master\Dump\\dumps\tatayoyo.obj.ldpdmp.tsv>
[11:37:42] [-] Unknown DN <CN=XXXXXXX,OU= XXX,OU=XXXX,OU=tatayoyo,DC=tatayoyo,DC=LOCAL>

[lbouillot]

Hi,

Thanks for your input. This issue is known and fixed in the future version of the tool we're still working on, in which the way ldap requests are handled has been completely changed.

Meanwhile, here are a few workaround you can try :

1. Use dsamain with a snapshot of your ntds.dit

   This is the preffered solution, since you'll be sure that there will be no modification during your dump. I know you already tried this, but make sure that :

   • You snapshot the real ntds.dit (the effective path is not always the default %SystemRoot%\System32\NTDS.dit, and is given by the registry key HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters)
   • You remount your snapshot with dsamain on the same Windows Server OS version that the one where you snapshotted it
   • You "recover" your database with esentutl /r edb /d [path] /i if you snapshotted it using vssadmin (no need for this if you're using ntdsutil with Windows Server 2008 or later)

2. Separate dump operations and retry only the one that fails

   Each dump operation done by LdapDump (ace, object, and schema) can be done independently.
   In your output, the Ace dump succeeded, but not the Object dump.
   You can plan to issue only this failing dump at a time you know there will be less modifications (for instance, at night).
   The default LdapDump command issued by the powershell script (LdapDump.exe -a <outfile_ace> -o <outfile_obj> -c <outfile_sch> ...) can be splitted in the following 3 commands :

   LdapDump.exe -A -a <outfile_ace> ...
   LdapDump.exe -O -o <outfile_obj> ...
   LdapDump.exe -C -c <outfile_sch> ...
   

3. Use this "quick'n'dirty" fix

   Instead of triggering a fatal error in Ldap.c:909 when this situation arrises, you could just display an error and continue to the next loop iteration. The result would be that the returned ldap entry would miss some attributes, and thus you could miss some control relations. This doesn't matter for deleted objects but this could be an issue for newly created objects. The following patch should do the job:

           if (idx == pLdapFullResults->dwResultCount) {
   -           FATAL(_T("Unknown DN <%s>"), pRetrievedResults[j]->tDN);
   +           LOG(Err, _T("Unknown DN <%s>"), pRetrievedResults[j]->tDN);
   +           continue;
           }
   

I hope this answer will be usefull to you. Thanks for using the tool.
Lucas.

[vletoux]

Hi,

Thanks for your feedback.
We managed to solve it by running it to a server close to the AD with 8 processors and 36 GB of memory. The main export did complete in less than 30 minutes (not the relations !).

About the image:
When tested on the local DC, mounting the dump with dsamain succeed while it failed on our offline workstation. We managed to fix the image by running esentutl -g and then -p. Then adding the -allowUpgrade switch to allow a read/write mount to fix the corrupted indexes.
But the export failed when the workstation seems to be out of memory (2GB of RAM with 16Go of swap).

The final result is very interesting (except that graphs are difficult to export as images !) but I think there are some data missing in the output. Maybe we can discuss it privately.

Vincent

[lbouillot]

Glad you found a workaround. Indeed, increasing the perfs of your environment was also a solution. Feel free to contact me using the email address on my github profile to discuss your graphs/missing data.