New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade: ldap data import too slow #361

Closed
mzico opened this Issue Oct 27, 2017 · 4 comments

Comments

Projects
None yet
4 participants
@mzico
Contributor

mzico commented Oct 27, 2017

I don't have much info / specific info on which DN/data are taking this much time but I'll do some tailing for import.log if possible.

Just to give an example: around 8800 entries ( ldap data + user's information ) import took 2 hours to complete.

@mzico mzico added the enhancement label Oct 27, 2017

@mzico mzico added this to the CE 3.2.0 milestone Oct 27, 2017

@arvindsinghtomar

This comment has been minimized.

Contributor

arvindsinghtomar commented Dec 4, 2017

As you can see there is too much process going on during export and import processes, scripts reads each of the data and modifies according to new servers configuration this process is definitely going to take some time .

I am not sure I will be able to make it fast or not .

But I will try whatever will be possible to reduce time.

@yurem

This comment has been minimized.

Contributor

yurem commented Dec 4, 2017

Can you try to use OpenLDAP .../sldapadd -b o=gluu -f file_name.ldif command to add entries?

@arvindsinghtomar

This comment has been minimized.

Contributor

arvindsinghtomar commented Dec 5, 2017

@yurem
'-b' is already included in commands

/opt/opendj/bin/ldapsearch -h localhost -p 1636 -Z -X -D "cn=directory manager" -j /tmp/tmpTl7cG9 -s one -b o=gluu o=* dn

@mbaser

This comment has been minimized.

Collaborator

mbaser commented Apr 6, 2018

Previously, upgrade scripts parses ldif files every time for each entry, this takes too long time for big data. Currently, ldif files parsed once and written to shelve database, this increases speed of upgrade script, importing 1e6 entries should not take more than 30 mins on a moderate server.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment