Create sync users #73

mvysny · 2018-05-23T13:30:29Z

Hi! Thank you very much for your awesome work. The DataGate is up and running, but I do not know how I can create sync users. I tried the following in the mongo console:

db.getCollection('no2.datagate.users').insert({ userName: "userId", password: "password", accountNonExpired: true, accountNonLocked: true, credentialsNonExpired: true, enabled: true });

Also I tried to create the users via the DataGate client moderation page, but the sync code still fails with

16:23:51.714 [ScheduledWorker.NO₂] ERROR org.dizitart.no2.sync.DataGateSyncTemplate - Remote error while getting online status
org.dizitart.no2.exceptions.SyncException: NO2.11020: HTTP/1.1 401 Unauthorized
	at org.dizitart.no2.sync.DataGateSyncTemplate.isOnline(DataGateSyncTemplate.java:180)
	at org.dizitart.no2.sync.SyncService.mergeChanges(SyncService.java:125)
	at org.dizitart.no2.sync.CollectionReplicator.run(CollectionReplicator.java:65)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Could you please paste an example how to create an user in DataGate, so that the sync client will work?

        val dataGateClient = DataGateClient("http://localhost:9090")
            .withAuth("sa", "sasasa")
        val syncTemplate = DataGateSyncTemplate(dataGateClient, "remote-collection@userId")

Also, what does the remote-collection@userId parameter mean?

The text was updated successfully, but these errors were encountered:

mvysny · 2018-05-23T13:33:26Z

Ok, it seems that

db.getCollection('no2.datagate.users').insert({ userName: "sa", password: "sasasa", accountNonExpired: true, accountNonLocked: true, credentialsNonExpired: true, enabled: true, authorities: ["USER"] });

did the trick. Now I'm getting 404 Not Found for the remote collection.

mvysny · 2018-05-23T14:08:26Z

Trying to create a remote user fails with 404:

        val dataGateClient = DataGateClient("http://localhost:9090")
            .withAuth("sa5", "sasasa")

        val userAccount = UserAccount()
        userAccount.userName = "sa6"
        userAccount.password = "sasasa"
        userAccount.authorities = arrayOf("USER")
        userAccount.collections = listOf("test@sa6")
//        DataGateUserTemplate(dataGateClient).getUserAccount("sa3")
        DataGateUserTemplate(dataGateClient).createRemoteUser(userAccount)

fails with:

Exception in thread "main" org.dizitart.no2.exceptions.SyncException: NO2.11003: HTTP/1.1 404 Not Found
	at org.dizitart.no2.sync.DataGateUserTemplate.createRemoteUser(DataGateUserTemplate.java:94)
	at MyMainKt.main(MyMain.kt:62)

The error message should be more specific and should provide more information to the cause of the error.

mvysny · 2018-05-23T14:09:07Z

I'm running DataGate 3.0.0 in Docker.

mvysny · 2018-05-23T14:14:28Z

I changed the port from 9090 to 8080 and now it seems to be doing sync. Unfortunately it fails with timeout exception:

17:14:06.028 [ScheduledWorker.NO₂] ERROR org.dizitart.no2.sync.DataGateSyncTemplate - Remote error while submitting change feed to remote
java.net.SocketTimeoutException: timeout
	at okio.Okio$4.newTimeoutException(Okio.java:230)
	at okio.AsyncTimeout.exit(AsyncTimeout.java:285)
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:241)
	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345)
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217)
	at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:212)
	at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at org.dizitart.no2.sync.DataGateClient$1.intercept(DataGateClient.java:154)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
	at okhttp3.RealCall.execute(RealCall.java:77)
	at org.dizitart.no2.sync.DataGateSyncTemplate.change(DataGateSyncTemplate.java:99)
	at org.dizitart.no2.sync.SyncService.mergeChanges(SyncService.java:235)
	at org.dizitart.no2.sync.SyncService.mergeChanges(SyncService.java:145)
	at org.dizitart.no2.sync.CollectionReplicator.run(CollectionReplicator.java:65)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketException: Socket closed
	at java.net.SocketInputStream.read(SocketInputStream.java:204)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at okio.Okio$2.read(Okio.java:139)
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
	... 33 common frames omitted
17:14:06.028 [ScheduledWorker.NO₂] ERROR org.dizitart.no2.sync.SyncService - Replication error
org.dizitart.no2.exceptions.SyncException: NO2.11011: error while submitting changes to remote collection
	at org.dizitart.no2.sync.DataGateSyncTemplate.change(DataGateSyncTemplate.java:109)
	at org.dizitart.no2.sync.SyncService.mergeChanges(SyncService.java:235)
	at org.dizitart.no2.sync.SyncService.mergeChanges(SyncService.java:145)
	at org.dizitart.no2.sync.CollectionReplicator.run(CollectionReplicator.java:65)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: timeout
	at okio.Okio$4.newTimeoutException(Okio.java:230)
	at okio.AsyncTimeout.exit(AsyncTimeout.java:285)
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:241)
	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345)
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217)
	at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:212)
	at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at org.dizitart.no2.sync.DataGateClient$1.intercept(DataGateClient.java:154)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
	at okhttp3.RealCall.execute(RealCall.java:77)
	at org.dizitart.no2.sync.DataGateSyncTemplate.change(DataGateSyncTemplate.java:99)
	... 10 common frames omitted
Caused by: java.net.SocketException: Socket closed
	at java.net.SocketInputStream.read(SocketInputStream.java:204)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at okio.Okio$2.read(Okio.java:139)
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
	... 33 common frames omitted

It seems as if the DataGate closed the connection. How can I configure the timeout to be bigger please?

anidotnet · 2018-05-23T15:12:58Z

Sorry for the inconvenience. Have you looked into the doc https://www.dizitart.org/nitrite-datagate.html
If you have already gone through it let me know I'll look into it.

mvysny · 2018-05-23T15:15:00Z

@anidotnet no problem, thank you very much for all of your work! Please ignore the user-related issues - I have solved them all out, so it is only enough to focus on the SocketTimeoutException. Yup I have looked at the documentation, unfortunately I could not find anything timeout-related.

anidotnet · 2018-05-23T15:16:44Z

Also, what does the remote-collection@userId parameter mean?

It means the copy of the remote-collection for user userid. Each user will have his/her own copy of the collection where they will sync.

anidotnet · 2018-05-23T15:23:31Z

For socket timeout check what is the port you have set for datagate server in dockerfile. Configure that port in DataGateClient. And also check what protocol are you using http or https because they use different ports.

The relevant portion you need to look into dockerfile is below

## Connection details (Replace with your own values)
ENV DATAGATE_HOST "0.0.0.0"
ENV DATAGATE_HTTP_PORT "8080"
ENV DATAGATE_HTTPS_PORT "8443"
ENV DATAGATE_MONITOR_PORT "9090"
ENV DATAGATE_KEY_STORE "keystore.jks"
ENV DATAGATE_KEY_PASSWORD "s3cret"

mvysny · 2018-05-23T15:25:19Z

Thanks! The port setting is ok - the sync client is apparently able to connect to DataGate and retrieve the information which documents have been synced. It apparently sees that there are no documents and starts sending all of local documents to the DataGate. However, it seems there are lots of local documents (100k) and it tries to upload them in one go and the whole process probably timeouts.

anidotnet · 2018-05-23T15:36:28Z

Currently I am traveling and commenting from my mobile. Once I get hold of my desktop, I'll look into the code and give you some insight. Meanwhile if possible share me the log file for datagate and nitrite.

anidotnet · 2018-05-23T17:36:48Z

Please try with setting larger read timeout inDataGateClient using readTimeout.

anidotnet · 2018-05-25T03:10:27Z

Any progress on it yet?

mvysny · 2018-05-25T05:58:19Z

Thank you @anidotnet and sorry for the delay. That setting indeed helped 👍 Now the client syncs with the DataGate properly. Yet it seems to be doing so at very low speed, estimating one record per second. In my testing database there is 10,000 records so I guess it will take some time :)

anidotnet · 2018-05-25T06:23:32Z

How heavy is your document? What is the network lag involved? During sync, nitrite will send all documents in one go.

mvysny · 2018-05-25T09:12:57Z

The document is really tiny, just one mapping from "animal" to "cat26"; it's just a testing database with 10000 of such tiny documents like cat1, cat2, ...

anidotnet · 2018-05-25T13:42:09Z

I am closing this issue now as the Socket timeout issue is resolved. Feel free to open another if you face any other issue.

mvysny · 2018-05-25T14:17:14Z

Of course. Thank you very much for your tip with the readTimeout

mvysny mentioned this issue May 23, 2018

Aedict 4: new data storage mvysny/aedict#834

Closed

anidotnet closed this as completed May 25, 2018

anidotnet mentioned this issue Aug 6, 2018

Nitrite DateGate not syncing #91

Closed

nitrite deleted a comment from dustin-auby Aug 6, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create sync users #73

Create sync users #73

mvysny commented May 23, 2018

mvysny commented May 23, 2018 •

edited

mvysny commented May 23, 2018

mvysny commented May 23, 2018

mvysny commented May 23, 2018 •

edited

anidotnet commented May 23, 2018

mvysny commented May 23, 2018 •

edited

anidotnet commented May 23, 2018

anidotnet commented May 23, 2018

mvysny commented May 23, 2018

anidotnet commented May 23, 2018

anidotnet commented May 23, 2018

anidotnet commented May 25, 2018

mvysny commented May 25, 2018 •

edited

anidotnet commented May 25, 2018

mvysny commented May 25, 2018 •

edited

anidotnet commented May 25, 2018

mvysny commented May 25, 2018

Create sync users #73

Create sync users #73

Comments

mvysny commented May 23, 2018

mvysny commented May 23, 2018 • edited

mvysny commented May 23, 2018

mvysny commented May 23, 2018

mvysny commented May 23, 2018 • edited

anidotnet commented May 23, 2018

mvysny commented May 23, 2018 • edited

anidotnet commented May 23, 2018

anidotnet commented May 23, 2018

mvysny commented May 23, 2018

anidotnet commented May 23, 2018

anidotnet commented May 23, 2018

anidotnet commented May 25, 2018

mvysny commented May 25, 2018 • edited

anidotnet commented May 25, 2018

mvysny commented May 25, 2018 • edited

anidotnet commented May 25, 2018

mvysny commented May 25, 2018

mvysny commented May 23, 2018 •

edited

mvysny commented May 23, 2018 •

edited

mvysny commented May 23, 2018 •

edited

mvysny commented May 25, 2018 •

edited

mvysny commented May 25, 2018 •

edited