Error in import30.py script

[zamilskhan]

OS: CentOS 6.8
Upgrade: 2.4.4 to 3.0.1
See ticket regarding the issue.
The following error is seen when the import command is used

GLUU.[root@centos ~]# ./import30.py backup_24/
INFO     Starting migration.
~~~~~~~ Gluu Server Community Edition Migartion Tool ~~~~~~~

Enter LDAP Server - 1.OpenLDAP, 2.OpenDJ [1]: 1
INFO     Stopping Webapps oxAuth and Identity.
INFO     Stopping OpenLDAP Server.
INFO     Copying the Certificates.
INFO     Updating the CA Certs Keystore.
WARNING  Error reading /etc/hostname
Traceback (most recent call last):
  File "./import30.py", line 599, in <module>
    migrator.migrate()
  File "./import30.py", line 582, in migrate
    self.copyCertificates()
  File "./import30.py", line 181, in copyCertificates
    hostname = self.readFile(self.etc_hostname).strip()
AttributeError: 'NoneType' object has no attribute 'strip'

[sahilIT2020]

Error on Ubuntu 14.04 when tried to upgrade 2.4.4 to 3.0.1
INFO Copying Shibboleth IDP files...
Traceback (most recent call last):
File "./import30.py", line 599, in
migrator.migrate()
File "./import30.py", line 584, in migrate
self.copyIDPFiles()
File "./import30.py", line 572, in copyIDPFiles
'/opt/shibboleth-idp/ssl')
File "/usr/lib/python2.7/distutils/dir_util.py", line 128, in copy_tree
"cannot copy tree '%s': not a directory" % src
distutils.errors.DistutilsFileError: cannot copy tree 'backup_24/opt/idp/ssl': not a directory

[tecoholic]

Commit 9fe6c81 has fix.

[zamilskhan]

@tecoholic

INFO     Importing Processed LDAP data.
"#############         67.84% eta    05s elapsed             11s spd  35.2 k/s slapadd: dn="inum=@!7B3A.F485.9E5E.AAC1!0001!8937.8C3B!0000!7506.EE20,ou=people,o=@!7B3A.F485.9E5E.AAC1!0001!8937.8C3B,o=gluu" (line=8804): (65) unrecognized objectClass 'ox-7B3AF4859E5EAAC1000189378C3B'
slapadd: dn="inum=@!7B3A.F485.9E5E.AAC1!0001!8937.8C3B!0000!2B6C.F252,ou=people,o=@!7B3A.F485.9E5E.AAC1!0001!8937.8C3B,o=gluu" (line=8820): (65) unrecognized objectClass 'ox-7B3AF4859E5EAAC1000189378C3B'
2017-03-31 08:48:31,752 ERROR    root Couldn't start the OpenLDAP server.
2017-03-31 08:48:31,753 ERROR    root Symas OpenLDAP LDAP services slapd starting^[[60G[^[[0;31mFAILED^[[0;39m]

When I try to start openldap

GLUU.[root@localhost ~]# service solserver start 
Symas OpenLDAP LDAP services slapd starting                [FAILED]

When Gluu Server is restarted

[root@centos ~]# service gluu-server-3.0.1 restart
Shutting down Gluu Server...
Stopping Apache...
OK
Stopping oxAuth RP...
OK
Stopping Asimba...
OK
Stopping CAS...
OK
Stopping IDP...
OK

Starting Gluu server, please wait...

[root@centos ~]# service gluu-server-3.0.1 login
gluu-server-3.0.1 is running...
logging in...
Welcome to the Gluu Server!
GLUU.[root@centos ~]# service solserver status
Symas OpenLDAP LDAP services slapd is not running

[tecoholic]

I figured out the issue. The ox-INUM object classes were deprecated during the 3.0 development and are no longer used. But the old data depends on the objectclass. I updated the script to concat the 99-user.ldif schema to the custom.schema of OpenLDAP in the commit 193e21d. This should fix the issue.

[zamilskhan]

@tecoholic this is from CentOS 6.8. The import does not throw any errors now, but oxTrust fails to start. When I restarted oxTrust and tailed log, I found the following error in the log.

oxtrust.zip

[tecoholic]

@zamilskhan PKIX path failure is usually due to the change in certificates. A restart of the gluu-server should reload the certificates and remove the error.

[tecoholic]

The last line in the upgrade docs https://gluu.org/docs/ce/3.0.1/upgrade/ says

After completion of import, stop/start 2.4.4 container one final time.

It feels as if it is hidden in the docs.

@jschristie Can you update the sentence to read "After completion of import, stop/start gluu-server container one time." as a Note box so that it is more visible and clear to developers.

[willow9886]

@tecoholic I just updated the docs accordingly. Thanks for the suggestion.

[tecoholic]

• /etc/hostname dependency has been completely removed. Uses the hostname CLI tool instead.
• import30.py doesn't start the webapps after import. Now it displays a message to restart the container.

Closing the issue as concerns were dealt with.

[zamilskhan]

Error from terminal from a CentOS 6.8 VM upgrade test from 2.4.4.2 to 3.0.1

ERROR    58f6e9f0 <= str2entry NULL (smr_normalize gluuLastUpdate 21)
slapadd: could not parse entry (line=9111)
58f6ea01 <= str2entry NULL (smr_normalize oxAuthAuthenticationTime 21)
slapadd: could not parse entry (line=14845)
58f6ea01 <= str2entry NULL (smr_normalize oxAuthAuthenticationTime 21)
slapadd: could not parse entry (line=14876)
58f6ea01 <= str2entry NULL (smr_normalize oxAuthAuthenticationTime 21)
slapadd: could not parse entry (line=14897)
Error, entries missing!
  entry 183: inum=@!2741.766b.6e81.a575!0002!d3de.0281,ou=appliances,o=gluu
  entry 184: ou=configuration,inum=@!2741.766b.6e81.a575!0002!d3de.0281,ou=appliances,o=gluu
  entry 293: ou=trustrelationships,inum=@!2741.766b.6e81.a575!0002!d3de.0281,ou=appliances,o=gluu

Restarted Gluu chroot, openID client missing.

[tecoholic]

Can you share the backup_24 folder as a zip file?

[tecoholic]

This issue was investigated again. Found no errors. It looks like the above issues have been sorted out after Shekar fixed the time parsing to the import script. Here is the output of testing.

------------------------------------------------------------
        Gluu Server Community Edition Migration Tool
============================================================

Enter LDAP Server - 1.OpenLDAP, 2.OpenDJ [1]: 1
INFO     Stopping Webapps oxAuth and Identity.
INFO     Stopping OpenLDAP Server.
INFO     Copying the Certificates.
INFO     Updating the CA Certs Keystore.
INFO     Copying the custom pages and assets of webapps.
INFO     Copying Shibboleth IDP files...
INFO     Converting Schema files of custom attributes.
INFO     Exporting LDAP data.
INFO     Processing the LDIF data.
INFO     Importing Processed LDAP data.
INFO     Fixing permissions for files.
INFO     Starting OpenLDAP Server.
============================================================
The migration is complete. Gluu Server needs to be restarted.


	# exit
	# service gluu-server-x.x.x restart

------------------------------------------------------------

However there is an important issue in CentOS 6.x:

The jsonmerge library that the import script depends on supports only Python 2.7 and not 2.6 which comes with the CentOS 6 by default. So in order to run the import sucessfully, the user will have to install Python2.7 and then run the script. Here are the instructions gathered from multiple sources.

1. Installing Python 2.7

# According to:
# How To Set Up Python 2.7.6 and 3.3.3 on CentOS 6.4
# https://www.digitalocean.com/community/tutorials/how-to-set-up-python-2-7-6-and-3-3-3-on-centos-6-4
yum -y update
yum groupinstall -y 'development tools'
yum install -y zlib-dev openssl-devel sqlite-devel bzip2-devel
yum install xz-libs
wget http://www.python.org/ftp/python/2.7.10/Python-2.7.10.tar.xz
xz -d Python-2.7.10.tar.xz
tar -xvf Python-2.7.10.tar
cd Python-2.7.10
./configure --prefix=/usr/local 
make
make altinstall
export PATH="/usr/local/bin:$PATH"

2. Installing Pip

curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
python2.7 get-pip.py

3. Installing JsonMerge

pip2.7 install jsonmerge

4. Running the import script

python2.7 import30.py backup_24