ldap authentication with latest version

[kforner]

Hi,

So on one of my server I can run tmapp. It works with the default Config.groovy. If I try to put the LDAP related bits from my current working config with the old instance, it does not seem to be taken into account (i.e. I can not login with the LDAP credentials).

If I just replace the Config.groovy with the old one, tmapp crashes. I can see these types of messages in the logs:

transmart.log:11-05-2017 17:48:21,118 ERROR GrailsContextLoader - Error initializing Grails: No bean named 'ldapAuthProvider' is defined
transmart.log:org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'ldapAuthProvider' is defined

So my question: do you have any information about how to setup LDAP authentication with this version, and if some thing has changed regarding the configuration ?

Thanks.
Karl

[dennyverbeeck]

Hi Karl,

The 'latest' branch is built from the code in the Transmart foundation's github repo. LDAP is an eTRIKS specific plugin and as far as i know it is not available in the foundation's version of Transmart. In a week or two we will have a brand new eTRIKS release. If possible for you i would suggest to wait until that release, as it will include all the latest developments from the Transmart Foundation's branch as well.

I'll drop a comment in this thread once the new release is up.

Thanks!
Denny

[kforner]

Ok. I also think it's best to wait then.
Are you using the LDAP plugin yourself, or testing it ?

[dennyverbeeck]

Ok, at the moment i'm not supporting any projects using it, nor am i using it myself. But if there are any issues we'll reach out to someone else within eTRIKS who has more knowledge on the LDAP plugin, and solve it together :)

[kforner]

cool !

[dennyverbeeck]

Hi Karl,

eTRIKS v4 has been released and i have built the new docker images. The master branch now contains this release. Keep in mind that, when logging in, you have to click the "Federated login" link below the login form to be redirected to your custom login service.
Looking forward to hear from you how it goes!

Denny

[kforner]

Great,
I'll test it ASAP, and don't worry you'll probably hear from me ;)
Thanks,
Karl

[hcountou]

Hello, great work !

Did you solved the Rserve issue that was not starting ?
I still get : "transmartdocker_tmrserve_1 exited with code 132", when composing with the new docker-compose you've provide. Any clue ?

If anyone has a successful procedure for setting the LDAP configuration right we are looking forward to it.

Hector

[dennyverbeeck]

Hi Hector,

Could you please provide the output of docker logs transmartdocker_tmrserve_1? I don't have an issue with Rserve on my system (Ubuntu 16.04).

Thanks!
Denny

[kforner]

same problem here:

> docker-compose up tmrserve
Creating network "transmartdocker_default" with the default driver
Creating transmartdocker_tmrserve_1
Attaching to transmartdocker_tmrserve_1
transmartdocker_tmrserve_1 exited with code 132

no logs:

docker logs transmartdocker_tmrserve_1

[hcountou]

docker logs transmartdocker_tmrserve_1 won't give me any logs if it isn't started...
We had to replace in previous version you tmrserve in order to make it work.

A collegue of mine did it. He made a new Dockerfile i'll ask him if i can forward it to you.

[dennyverbeeck]

Interesting, i can confirm on an ubuntu 14.04 the same issue. I know Karl is running that, is that what you are running as well Hector?

I see from the dockerfile you are getting the 16.1 version of transmart-data. It might be the case that having an older version will not install all required R packages (don't have time to check right now). This would impact things like SmartR and the advanced analysis workflows.

In any case, building on a 14.04 system with the original Dockerfile also results in an image that runs on both 14.04 and 16.04, while the original image, built on a 16.04 system, does not run on 14.04. My best guess is that since there is a difference in kernel versions, there might be a difference in library versions being linked to when compiling R.

The new image is now pushed to dockerhub. You should be able to pull it by just doing a docker pull dennyverbeeck/transmart-rserve:etriks-v4.0. Then the regular docker-compose up should pick up the newly pulled image 👍

[hcountou]

Yes we are still on Ubuntu 14.04. I'll test your new version right away. And tell you the conclusion.

Thanks.

[kforner]

not better:

> docker pull dennyverbeeck/transmart-rserve:etriks-v4.0
etriks-v4.0: Pulling from dennyverbeeck/transmart-rserve
Digest: sha256:d3771e83915b7da509e87ebcabb62f00ddb92c8a657c8921c9c71e2af15f648b
Status: Image is up to date for dennyverbeeck/transmart-rserve:etriks-v4.0

> docker-compose  up tmrserve
Creating network "transmartdocker_default" with the default driver
Creating transmartdocker_tmrserve_1
Attaching to transmartdocker_tmrserve_1
transmartdocker_tmrserve_1 exited with code 132

[hcountou]

Same here. Still got the error.

[dennyverbeeck]

Strange, could you give me the output of docker version and docker-compose version? In the meantime, probably the safest way to move forward for is for you to build the image yourself locally. You can just run docker build -t dennyverbeeck/transmart-rserve:etriks-v4.0 . in the transmart-rserve directory to build it and overwrite the image from dockerhub. Alternatively, you can provide your own image tag after the -t flag but then you need to update your docker-compose.yml file so that the same tag is used there as well.

[kforner]

>docker version 
Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:06:06 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:06:06 2017
 OS/Arch:      linux/amd64
 Experimental: false

> docker-compose version
docker-compose version 1.11.2, build dfed245
docker-py version: 2.1.0
CPython version: 2.7.13
OpenSSL version: OpenSSL 1.0.1t  3 May 2016

[kforner]

building the local rserve container works for me (but it takes ages).
I end up with a transmart that seems operational.
I'll try a LDAP config.

[kforner]

Does not work, and the error messages look familiar:

No bean named 'ldapAuthProvider' is defined

root@5f0da3a06a98:/usr/local/tomcat/logs# more localhost.2017-05-30.log 
May 30, 2017 4:20:41 PM org.apache.catalina.core.ApplicationContext log
INFO: No Spring WebApplicationInitializer types detected on classpath
May 30, 2017 4:20:43 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
May 30, 2017 4:21:09 PM org.apache.catalina.core.StandardContext listenerStart
SEVERE: Exception sending context initialized event to listener instance of class org.codehaus.groovy.grails.web.context.GrailsContextLoaderListener
org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'ldapAuthProvider' is defined
	at SpringSecurityCoreGrailsPlugin$_createBeanList_closure22.doCall(SpringSecurityCoreGrailsPlugin.groovy:795)
	at SpringSecurityCoreGrailsPlugin.createBeanList(SpringSecurityCoreGrailsPlugin.groovy:795)
	at SpringSecurityCoreGrailsPlugin$_closure4.doCall(SpringSecurityCoreGrailsPlugin.groovy:700)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

[dennyverbeeck]

Hi Karl, did you do a docker-compose down -v? Otherwise the volumes from previous containers will get mounted in the new containers, so the volume from the transmart-app:latest container will get mounted in the transmart-app:etriks-v4.0 container and thus the new container will not extract it's war file on first startup, leaving you with the same old version of the application.

PS: when you want to upgrade the tmapp container in the future without removing also the database volumes, you can just remove the transmartdocker_appwebapps volume. These instruction were already on the latest branch but i just merged them to master as well here.

[hcountou]

docker version
Client:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built: Wed Oct 26 21:44:32 2016
OS/Arch: linux/amd64

Server:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built: Wed Oct 26 21:44:32 2016
OS/Arch: linux/amd64

docker-compose version
docker-compose version 1.11.2, build dfed245
docker-py version: 2.1.0
CPython version: 2.7.6
OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014

[dennyverbeeck]

Thanks both for the replies, i have a system with the exact same versions of ubuntu, docker and compose as Karl but can not reproduce the rserve error. According to this post it's possible that my CPU where R was compiled, has instruction sets not available in your CPU. But i don't even know for sure which instruction sets are called by R. However it's starting to seem like one of the few remaining possibilities :p I could check if you gave me a cat /proc/cpuinfo, but it would be difficult to confirm anyway. Rebuilding the image on the host system seems to be the most appropriate solution for now.

Any luck with LDAP? In the meantime i've reached out within eTRIKS to see who is familiar with the LDAP configuration.

[kforner]

I have done the docker-compose down -v. Same result:
ERROR GrailsContextLoader - Error initializing the application: No bean named 'ldapAuthProvider' is defined
It seems that the LDAP plugin is still missing.
Denny, you has reasons to think it would be present in that release ? How so ?
Thanks.

[dennyverbeeck]

This exact same version (in fact the same war file) is running on an eTRIKS server, where we can log in with our central etriks credentials, through LDAP. So it should be available. I'll get back to you once i know more about the configuration.

[kforner]

ok, so it might be my config. I'll check.

[dennyverbeeck]

Hi Karl,

Just adding grails.plugin.springsecurity.ldap.active = true to Config.groovy seems to enable LDAP:

...
tmapp_1     | [INFO] Including configuration file [file:/root/.grails/transmartConfig/Config.groovy] in configuration building.
tmapp_1     | [INFO] Including configuration file [file:/root/.grails/transmartConfig/DataSource.groovy] in configuration building.
tmapp_1     |
tmapp_1     | Configuring Spring Security Core ...
tmapp_1     | ... finished configuring Spring Security Core
tmapp_1     |
tmapp_1     | Configuring Spring Security OAuth2 provider ...
tmapp_1     | ... done configuring Spring Security OAuth2 provider
tmapp_1     |
tmapp_1     | Configuring Spring Security LDAP ...
tmapp_1     | ... finished configuring Spring Security LDAP
tmapp_1     |
tmapp_1     | Configuring XNAT resources ...
...

I haven't heard back yet from my colleagues in etriks, but presumably you should use the configuration options listed on the Spring security LDAP grails plugin (version 2.0) manual here. I currently dont have an ldap server to test this against though.

[kforner]

right, adding

grails.plugin.springsecurity.ldap.active = true

solves the problem. Transmart launches, but I still can not connect using LDAP, but that's probably my config.
Although I do not see the "Federated login" link you mentioned... Is this normal ?

[dennyverbeeck]

Yes, sorry about that! I was confusing the SAML login with LDAP login. LDAP uses the regular login form and connects to the ldap server behind the scenes. SAML is if you have a single sign on service with which you want to integrate. Therefore it needs this extra link to redirect you to the single sign on server.

[kforner]

Trying to debug the LDAP: when trying to connect, I get this in transmart.log:

31-05-2017 12:40:53,566 DEBUG DefaultLdapAuthoritiesPopulator - Roles from search: [transmart_role_admin]
31-05-2017 12:40:54,936 ERROR JDBCExceptionReporter - Batch entry 0 insert into SEARCH_AUTH_PRINCIPAL (DATE_CREATED, DESCRIPTION, ENABLED, LAST_UPDATED, NAME, PRINCIPAL_TYPE, UNIQUE_ID, ID) values ('2017-05-31 12:40:54.931000 +00:00:00', '', '1', '2017-05-31 12:40:54.931000 +00:00:00', 'Karl Forner', 'USER', '', 2) was aborted.  Call getNextException to see the cause.
31-05-2017 12:40:54,937 ERROR JDBCExceptionReporter - ERROR: duplicate key value violates unique constraint "pk_search_principal"
  Detail: Key (id)=(2) already exists.
31-05-2017 12:40:54,941 ERROR PatchedDefaultFlushEventListener - Could not synchronize database state with session
org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
	at com.recomdata.security.LdapAuthUserDetailsMapper.findOrSaveUser(LdapAuthUserDetailsMapper.groovy:111)
	at com.recomdata.security.LdapAuthUserDetailsMapper$_mapUserFromContext_closure5.doCall(LdapAuthUserDetailsMapper.groovy:151)
	at org.grails.datastore.gorm.GormStaticApi.withTransaction(GormStaticApi.groovy:815)
	at org.grails.datastore.gorm.GormStaticApi.withTransaction(GormStaticApi.groovy:715)
	at com.recomdata.security.LdapAuthUserDetailsMapper.mapUserFromContext(LdapAuthUserDetailsMapper.groovy:150)
	at grails.plugin.springsecurity.web.authentication.RequestHolderAuthenticationFilter.attemptAuthentication(RequestHolderAuthenticationFilter.java:80)
	at grails.plugin.springsecurity.web.authentication.RequestHolderAuthenticationFilter.doFilter(RequestHolderAuthenticationFilter.java:53)
	at grails.plugin.springsecurity.web.authentication.logout.MutableLogoutFilter.doFilter(MutableLogoutFilter.java:62)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.BatchUpdateException: Batch entry 0 insert into SEARCH_AUTH_PRINCIPAL (DATE_CREATED, DESCRIPTION, ENABLED, LAST_UPDATED, NAME, PRINCIPAL_TYPE, UNIQUE_ID, ID) values ('2017-05-31 12:40:54.931000 +00:00:00', '', '1', '2017-05-31 12:40:54.931000 +00:00:00', 'Karl Forner', 'USER', '', 2) was aborted.  Call getNextException to see the cause.
	at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2762)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1891)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2909)
	... 11 more
31

additionally, and probably not related, there are numerous errors about Rserve:

31-05-2017 12:41:38,844  WARN RScriptsSynchronizer - Synchronization of R scripts has failed. Retries left: 7
org.rosuda.REngine.Rserve.RserveException: Cannot connect: Connection refused (Connection refused)
   at org.rosuda.REngine.Rserve.RConnection.<init>(RConnection.java:88)
   at org.rosuda.REngine.Rserve.RConnection.<init>(RConnection.java:60)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
   at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
   at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
   at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:194)
   at heim.rserve.RConnectionProvider.get(RConnectionProvider.groovy:16)
   at heim.rserve.RConnectionProvider$get.call(Unknown Source)
   at heim.rserve.RScriptsSynchronizer.copyFilesOver(RScriptsSynchronizer.groovy:78)
   at heim.rserve.RScriptsSynchronizer.this$2$copyFilesOver(RScriptsSynchronizer.groovy)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
   at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
   at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1086)
   at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)
   at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:910)
   at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1031)
   at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)
   at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:910)
   at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
   at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:137)
   at heim.rserve.RScriptsSynchronizer$_start_closure1.doCall(RScriptsSynchronizer.groovy:48)
   at heim.rserve.RScriptsSynchronizer$_start_closure1.doCall(RScriptsSynchronizer.groovy)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
   at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
   at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1086)
   at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)
   at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:910)
   at groovy.lang.Closure.call(Closure.java:411)
   at groovy.lang.Closure.call(Closure.java:405)
   at groovy.lang.Closure.run(Closure.java:492)
   at java.lang.Thread.run(Thread.java:745)

[dennyverbeeck]

Regarding rserve, the transmart error says it cannot connect to rserve. If the rserve container is running, and is on the transmartdocker_default docker network (it should be if it's managed by docker-compose), then it should be accessible. Could you confirm this by running docker inspect transmartdocker_tmrserve_1 | grep -A 7? The output should look like this:

$ docker inspect transmartdocker_tmrserve_1 | grep -A 7 Networks
            "Networks": {
                "transmartdocker_default": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "336bc908b526",
                        "tmrserve"
                    ],

To directly test if the rserve container is available from the tmapp container:

$ docker exec -it transmartdocker_tmapp_1 ping -c 4 tmrserve
PING tmrserve (172.18.0.2): 56 data bytes
64 bytes from 172.18.0.2: icmp_seq=0 ttl=64 time=0.096 ms
64 bytes from 172.18.0.2: icmp_seq=1 ttl=64 time=0.103 ms
64 bytes from 172.18.0.2: icmp_seq=2 ttl=64 time=0.173 ms
64 bytes from 172.18.0.2: icmp_seq=3 ttl=64 time=0.145 ms
--- tmrserve ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.096/0.129/0.173/0.031 ms

[kforner]

Yes it pings. And the /transmart-data/R/root/lib/R/bin/Rserve.dbg process is running in tmrserve.

[kforner]

solved the rserve problem, was due to my Config.groovy, sorry.

[dennyverbeeck]

Great you found it! 😄 no problem.

In the meantime i've had some responses regarding LDAP. In attached file tm_struct.ldif you can find the groups necessary for transmart to be in ldap. cnorris.ldif an example transmart user in LDAP, and in tmconfig.txt the relevant config section to put in Config.groovy.

tmconfig.txt
tm_struct.ldif.txt
cnorris.ldif.txt

[kforner]

I tried to mix my config with your concerning authentication/authorization, but it still fails.

The crux seems to be:
31-05-2017 15:20:06,117 ERROR JDBCExceptionReporter - ERROR: duplicate key value violates unique constraint "pk_search_principal"

My ldap config seems to work:

31-05-2017 15:20:04,512 DEBUG BindAuthenticator - Attempting to bind as uid=karl,cn=users,cn=accounts,dc=quartzbio,dc=com
31-05-2017 15:20:04,517 DEBUG BindAuthenticator - Retrieving attributes...
31-05-2017 15:20:04,525 DEBUG DefaultLdapAuthoritiesPopulator - Getting authorities for user uid=karl,cn=users,cn=accounts,dc=quartzbio,dc=com

so it found me

31-05-2017 15:20:04,526 DEBUG DefaultLdapAuthoritiesPopulator - Searching for roles for user 'karl', DN = 'uid=karl,cn=users,cn=accounts,dc=quartzbio,dc=com', with filter (&(cn=transmart_role*)(member={0})) in search base 'cn=groups,cn=accounts,dc=quartzbio,dc=com'
31-05-2017 15:20:04,526 DEBUG SpringSecurityLdapTemplate - Using filter: (&(cn=transmart_role*)(member=uid=karl,cn=users,cn=accounts,dc=quartzbio,dc=com))
31-05-2017 15:20:04,537 DEBUG DefaultLdapAuthoritiesPopulator - Roles from search: [transmart_role_admin]

it even found a role: transmart_role_admin

31-05-2017 15:20:06,117 ERROR JDBCExceptionReporter - Batch entry 0 insert into SEARCH_AUTH_PRINCIPAL (DATE_CREATED, DESCRIPTION, ENABLED, LAST_UPDATED, NAME, PRINCIPAL_TYPE, UNIQUE_ID, ID) values ('2017-05-31 15:20:06.088000 +00:00:00', '', '1', '2017-05-31 15:20:06.088000 +00:00:00', 'Karl Forner', 'USER', '', 1) was aborted.  Call getNextException to see the cause.

but it cannot insert this in the DB

31-05-2017 15:20:04,525 DEBUG DefaultLdapAuthoritiesPopulator - Getting authorities for user uid=karl,cn=users,cn=accounts,dc=quartzbio,dc=com
31-05-2017 15:20:04,526 DEBUG DefaultLdapAuthoritiesPopulator - Searching for roles for user 'karl', DN = 'uid=karl,cn=users,cn=accounts,dc=quartzbio,dc=com', with filter (&(cn=transmart_role*)(member={0})) in search base 'cn=groups,cn=accounts,dc=quartzbio,dc=com'
31-05-2017 15:20:04,526 DEBUG SpringSecurityLdapTemplate - Using filter: (&(cn=transmart_role*)(member=uid=karl,cn=users,cn=accounts,dc=quartzbio,dc=com))
31-05-2017 15:20:04,537 DEBUG DefaultLdapAuthoritiesPopulator - Roles from search: [transmart_role_admin]
31-05-2017 15:20:06,117 ERROR JDBCExceptionReporter - Batch entry 0 insert into SEARCH_AUTH_PRINCIPAL (DATE_CREATED, DESCRIPTION, ENABLED, LAST_UPDATED, NAME, PRINCIPAL_TYPE, UNIQUE_ID, ID) values ('2017-05-31 15:20:06.088000 +00:00:00', '', '1', '2017-05-31 15:20:06.088000 +00:00:00', 'Karl Forner', 'USER', '', 1) was aborted.  Call getNextException to see the cause.
31-05-2017 15:20:06,117 ERROR JDBCExceptionReporter - ERROR: duplicate key value violates unique constraint "pk_search_principal"
  Detail: Key (id)=(1) already exists.
31-05-2017 15:20:06,125 ERROR PatchedDefaultFlushEventListener - Could not synchronize database state with session
org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
	at com.recomdata.security.LdapAuthUserDetailsMapper.findOrSaveUser(LdapAuthUserDetailsMapper.groovy:111)
	at com.recomdata.security.LdapAuthUserDetailsMapper$_mapUserFromContext_closure5.doCall(LdapAuthUserDetailsMapper.groovy:151)
	at org.grails.datastore.gorm.GormStaticApi.withTransaction(GormStaticApi.groovy:815)
	at org.grails.datastore.gorm.GormStaticApi.withTransaction(GormStaticApi.groovy:715)
	at com.recomdata.security.LdapAuthUserDetailsMapper.mapUserFromContext(LdapAuthUserDetailsMapper.groovy:150)
	at grails.plugin.springsecurity.web.authentication.RequestHolderAuthenticationFilter.attemptAuthentication(RequestHolderAuthenticationFilter.java:80)
	at grails.plugin.springsecurity.web.authentication.RequestHolderAuthenticationFilter.doFilter(RequestHolderAuthenticationFilter.java:53)
	at grails.plugin.springsecurity.web.authentication.logout.MutableLogoutFilter.doFilter(MutableLogoutFilter.java:62)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.BatchUpdateException: Batch entry 0 insert into SEARCH_AUTH_PRINCIPAL (DATE_CREATED, DESCRIPTION, ENABLED, LAST_UPDATED, NAME, PRINCIPAL_TYPE, UNIQUE_ID, ID) values ('2017-05-31 15:20:06.088000 +00:00:00', '', '1', '2017-05-31 15:20:06.088000 +00:00:00', 'Karl Forner', 'USER', '', 1) was aborted.  Call getNextException to see the cause.
	at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2762)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1891)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2909)
	... 11 more
31-05-2017 15:20:06,132 DEBUG HttpSessionSecurityContextRepository - SecurityContext is empty or contents are anonymous - context will not be stored in HttpSession.
31-05-2017 15:20:06,132 DEBUG SecurityContextPersistenceFilter - SecurityContextHolder now cleared, as request processing completed

[kforner]

my config seems quite close to tmconfig.txt, I disabled oauth and saml:

grails.plugin.springsecurity.providerNames =  ['ldapAuthProvider']
grails { plugin { springsecurity {

  ldap {
    active = true
    context {
        // Address of the LDAP server
        server = 'ldap://xxxxxxx'
        // Distinguished Name (DN) to authenticate with
        managerDn = 'uid=xxxxxxxx,cn=users,cn=accounts,dc=quartzbio,dc=com'
        // Password to authenticate with
        managerPassword = 'xxxxxxxx'
        // Whether or not integrate internal roles. Blurry notion, documentation missing
//        allowInternaRoles = false
    }

    search {
        // Context name to search in, relative to the base of the configured ContextSource
        base = 'cn=users,cn=accounts,dc=quartzbio,dc=com'
        // The filter expression used in the user search: this how the authorization is implemented:
        // must be member of app-transmart group
        filter = '(&(uid={0})(memberOf=cn=app-transmart,cn=groups,cn=accounts,dc=quartzbio,dc=com))'
    }
    authenticator {
       // Names of attribute ids to return; use null to return all and an empty list to return none
       authenticator.attributesToReturn = ['uid', 'mail', 'cn']
       useBind = true
    }

 
    // Mapping attribute for your LDAP user details
    mapper.userDetailsClass = 'person'

    authorities {
        // The base DN from which the search for group membership should be performed
        groupSearchBase = 'cn=groups,cn=accounts,dc=quartzbio,dc=com'
        // The pattern to be used for the user search. {0} is the user's DN
        // here we consider only the groups like transmart_role having the user as member
	groupSearchFilter = '(&(cn=transmart_role*)(member={0}))'
//        groupSearchFilter = 'member={0}'
        // The ID of the attribute which contains the role name for a group
        groupRoleAttribute = 'cn'
	// An optional string prefix to strip from the beginning of LDAP group names. For example, 'EnHS-' will change EnHS-Staff-All to ROLE_Staff-All
	clean.uppercase = true
	clean.prefix = 'TRANSMART_ROLE_'

        // Whether PartialResultExceptions should be ignored in searches, typically used with Active Directory since AD servers often have a problem with referrals
        ignorePartialResultException = true
        // Whether to infer roles based on group membership
        retrieveGroupRoles = true

        // Whether to retrieve additional roles from the database using the User/Role many-to-many
        retrieveDatabaseRoles = false
    }

} } } }

[hcountou]

"rserve Config.groovy problem", what kind of problem exactly ?

[kforner]

@hcountou:
RModules.host = "tmrserve"
RModules.port = 6311

my host was localhost, based on my existing config, which does not use linked (or networked) containers.

[dennyverbeeck]

Ok so we're getting closer! The default transmart installation comes with a number of preconfigured users, admin, guest, and a few more. It seems it wants to insert the ldap user into there as well, but for some reason takes an existing id. I'm not sure if this is expected behavior. I'll reach out to another developer in etriks.

[kforner]

ok, makes sense. Because I configured it to only use ldap provider, it must not load users already in DB...

[dennyverbeeck]

Hi Karl,

So, in the case of LDAP authentication, transmart still stores the credentials in its own database, since it relies on this for permissions grouping and so on. TM will keep authenticating against LDAP, but will always update its local profile of the user.

In the meantime i've set up an openldap locally to test this out, and also encountered the same issue you encountered. TM comes with a few accounts pre-configured (admin, guest, etc). The reason for our issue is that the mapping logic does not correctly take into account users already stored in the database, and as a result tries to store the LDAP user with a primary key that's already in use. This is a bug and we will resolve it in a future release. However, since you are only using LDAP for authentication, and no local accounts, we can safely remove the preconfigured users from the database using:

$ docker exec -it transmartdocker_tmdb_1 psql transmart -c 'truncate searchapp.search_auth_principal cascade;'
NOTICE:  truncate cascades to table "search_auth_user"
NOTICE:  truncate cascades to table "search_auth_group"
NOTICE:  truncate cascades to table "search_auth_group_member"
NOTICE:  truncate cascades to table "search_auth_sec_object_access"
NOTICE:  truncate cascades to table "search_gene_signature"
NOTICE:  truncate cascades to table "saved_faceted_search"
NOTICE:  truncate cascades to table "search_auth_user_sec_access"
NOTICE:  truncate cascades to table "search_role_auth_user"
TRUNCATE TABLE

When you try to login now, it should be successful. See an excerpt from the logs on my local instance logging in after clearing out the local transmart users:

2017-06-01 10:11:46,804 [ajp-apr-8009-exec-10] DEBUG ldap.SpringSecurityLdapTemplate  - Found DN: cn=Denny Verbeeck,ou=Users,dc=etriks,dc=eu
2017-06-01 10:11:46,805 [ajp-apr-8009-exec-10] DEBUG authentication.BindAuthenticator  - Attempting to bind as cn=Denny Verbeeck,ou=Users,dc=etriks,dc=eu
2017-06-01 10:11:46,807 [ajp-apr-8009-exec-10] DEBUG authentication.BindAuthenticator  - Retrieving attributes...
2017-06-01 10:11:46,808 [ajp-apr-8009-exec-10] DEBUG userdetails.DefaultLdapAuthoritiesPopulator  - Getting authorities for user cn=Denny Verbeeck,ou=Users,dc=etriks,dc=eu
2017-06-01 10:11:46,809 [ajp-apr-8009-exec-10] DEBUG userdetails.DefaultLdapAuthoritiesPopulator  - Searching for roles for user 'denny', DN = 'cn=Denny Verbeeck,ou=Users,dc=etriks,dc=eu', with filter memberUid={1} in search base 'ou=tra
nsmart,dc=etriks,dc=eu'
2017-06-01 10:11:46,809 [ajp-apr-8009-exec-10] DEBUG ldap.SpringSecurityLdapTemplate  - Using filter: memberUid=denny
2017-06-01 10:11:46,810 [ajp-apr-8009-exec-10] DEBUG userdetails.DefaultLdapAuthoritiesPopulator  - Roles from search: []
2017-06-01 10:11:46,810 [ajp-apr-8009-exec-10] DEBUG security.LdapAuthUserDetailsMapper  - Mapping user details from context and database with username: denny
2017-06-01 10:11:48,450 [ajp-apr-8009-exec-10] DEBUG security.BruteForceLoginLockService  - successfull login for denny

After this, additional ldap logins get added without a problem to the local database as well, so you should be good to go!

Hope this helps 😄

Denny

[kforner]

Brilliant ! Works perfectly !!!
Thanks a lot Denny for your excellent support and reactivity.
Now that my main prerequisite (LDAP) is working, I'll tweak the docker-compose.yml to suit my needs:

• have the database use a local dir instead of a volume
• same for logs
• export the DB port

The transmart app is to be under the umbrella of an apache reverse-proxy of mine. Should I disable tmweb ?

[dennyverbeeck]

Great!!

For the database: i can point you to this volume driver. I've used it before. When you've installed it, it allows you to create a named volume that is mounted on a specified location in your host system, like this: docker volume create -d local-persist -o mountpoint=/data/transmartdb --name=transmartdb. Or you can do everything from the docker-compose file by modifying the volumes section:

volumes:
  appdata:
    driver: local-persist
    driver_opts:
      mountpoint: "/mnt/temporal/transmart-data"
  apptmp:
    driver: local-persist
    driver_opts:
      mountpoint: "/mnt/temporal/transmart-tmp"
  appwebapps:
  postgresdata:
    driver: local-persist
    driver_opts:
      mountpoint: "/mnt/db/data"
  postgreslogs:
    driver: local-persist
    driver_opts:
      mountpoint: "/mnt/db/logs"

The DB port is exported by the default compose file, but only to localhost, for security reasons. If you want to expose it to remote hosts as well, just remove the 127.0.0.1 loopback ip address from the tmdb ports section. It will cause the port to be bound to all interfaces.

Finally, yes you can safely disable the tmweb service. If your apache reverse proxy is on a different host, you'll need to also remove the 127.0.0.1 part of tmapp's ports section to expose it to remote hosts. If it's on the same host, you can leave it as is.

Good luck!