Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System collection support (and more) #19

Closed
okapies opened this issue May 28, 2015 · 6 comments
Closed

System collection support (and more) #19

okapies opened this issue May 28, 2015 · 6 comments
Assignees
@gortiz

Comments

@okapies
Copy link

okapies commented May 28, 2015

ToroDB throws errors while access to Mongo's system collections, whose identifier is prefixed with system. (Mongoose ODM, used in some applications like Let's Chat, utilizes them).

20489 [torod-session-3] DEBUG c.t.t.d.p.query.QueryEvaluator - Query (migrationId == 1421181022156-drop-sessions) or (migrationId exists ( == 1421181022156-drop-sessions)) fulfiled by [] 
20490 [nioEventLoopGroup-3-5] DEBUG c.e.m.m.RequestMessageObjectHandler - Received message type: OP_INSERT, data: InsertMessage{clientAddress=127.0.0.1, clientPort=57308, requestId=0, database='letschat', collection='system.indexes', documents=[{ "ns" : "letschat.rooms" , "key" : { "slug" : 1} , "name" : "slug_1" , "unique" : true , "background" : true , "safe" :  null }]} 
20490 [nioEventLoopGroup-3-4] DEBUG c.e.m.m.RequestMessageObjectHandler - Received message type: OP_INSERT, data: InsertMessage{clientAddress=127.0.0.1, clientPort=57307, requestId=0, database='letschat', collection='system.indexes', documents=[{ "ns" : "letschat.users" , "key" : { "email" : 1} , "name" : "email_1" , "unique" : true , "background" : true , "safe" :  null }]} 
20493 [nioEventLoopGroup-3-4] DEBUG c.t.t.db.metaInf.CollectionMetaInfo - Heuristic said 50001 new ids are needed. Difference between lastCachedId and lastUsedId is -1 
20493 [nioEventLoopGroup-3-5] DEBUG c.t.t.db.metaInf.CollectionMetaInfo - Heuristic said 50002 new ids are needed. Difference between lastCachedId and lastUsedId is -2 
20493 [torod-system-1] ERROR c.t.t.d.e.DefaultExecutorFactory - System executor exception 
com.torodb.torod.core.exceptions.ToroRuntimeException: java.lang.IllegalArgumentException: At the present time Torod doesn't support '.' as identifier character. Only a alphanumeric letters and '_' are supported
        at com.torodb.torod.db.executor.jobs.SystemDbCallable.onFail(SystemDbCallable.java:64) ~[torodb.jar:na]
        at com.torodb.torod.db.executor.jobs.Job.call(Job.java:22) ~[torodb.jar:na]
        at com.torodb.torod.db.executor.DefaultSystemExecutor$SystemRunnable.call(DefaultSystemExecutor.java:153) ~[torodb.jar:na]
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.7.0_25]
        at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.7.0_25]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_25]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_25]
        at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
Caused by: java.lang.IllegalArgumentException: At the present time Torod doesn't support '.' as identifier character. Only a alphanumeric letters and '_' are supported
        at com.torodb.torod.db.postgresql.IdsFilter.filter(IdsFilter.java:42) ~[torodb.jar:na]
        at com.torodb.torod.db.postgresql.IdsFilter.escapeSchemaName(IdsFilter.java:30) ~[torodb.jar:na]
        at com.torodb.torod.db.sql.AbstractSqlDbConnection.createCollection(AbstractSqlDbConnection.java:152) ~[torodb.jar:na]
        at com.torodb.torod.db.executor.jobs.CreateCollectionCallable.call(CreateCollectionCallable.java:61) ~[torodb.jar:na]
        at com.torodb.torod.db.executor.jobs.CreateCollectionCallable.call(CreateCollectionCallable.java:34) ~[torodb.jar:na]
        at com.torodb.torod.db.executor.jobs.SystemDbCallable.failableCall(SystemDbCallable.java:72) ~[torodb.jar:na]
        at com.torodb.torod.db.executor.jobs.Job.call(Job.java:20) ~[torodb.jar:na]
        ... 6 common frames omitted

ToroDB also complains about upper case characters:

9230 [torod-system-1] DEBUG org.jooq.tools.LoggerListener - Executing query          : insert into "torodb"."collections" ("name", "schema", "capped", "max_size", "max_elementes", "other", "storage_engine") values (?, ?, ?, ?, ?, ?, ?) 
9230 [torod-system-1] DEBUG org.jooq.tools.LoggerListener - -> with bind values      : insert into "torodb"."collections" ("name", "schema", "capped", "max_size", "max_elementes", "other", "storage_engine") values ('migrootions', 'migrootions', false, 0, 0, null, 'torodb-dev') 
9233 [torod-system-1] DEBUG org.jooq.tools.LoggerListener - Affected row(s)          : 1 
9233 [torod-system-1] DEBUG org.jooq.tools.StopWatch - Query executed           : Total: 3.934ms 
9233 [torod-system-1] DEBUG org.jooq.tools.StopWatch - Finishing                : Total: 4.001ms, +0.067ms 
9272 [torod-system-1] DEBUG org.jooq.tools.LoggerListener - Executing query          : select count(*) from information_schema.tables where (table_schema = ? and table_name = ?) 
9273 [torod-system-1] DEBUG org.jooq.tools.LoggerListener - -> with bind values      : select count(*) from information_schema.tables where (table_schema = 'migrootions' and table_name = 'indexes') 
9276 [torod-system-1] DEBUG org.jooq.tools.StopWatch - Query executed           : Total: 7.332ms 
9277 [torod-system-1] DEBUG org.jooq.tools.LoggerListener - Fetched result           : +-----+ 
9277 [torod-system-1] DEBUG org.jooq.tools.LoggerListener -                          : |count| 
9277 [torod-system-1] DEBUG org.jooq.tools.LoggerListener -                          : +-----+ 
9277 [torod-system-1] DEBUG org.jooq.tools.LoggerListener -                          : |    0| 
9277 [torod-system-1] DEBUG org.jooq.tools.LoggerListener -                          : +-----+ 
9277 [torod-system-1] DEBUG org.jooq.tools.StopWatch - Finishing                : Total: 7.864ms, +0.531ms 
9278 [torod-system-1] DEBUG org.jooq.tools.LoggerListener - Executing query          : CREATE TABLE migrootions.indexes (name VARCHAR PRIMARY KEY, index JSONB NOT NULL) 
9279 [nioEventLoopGroup-3-4] DEBUG c.t.t.db.metaInf.CollectionMetaInfo - migrootions.(_id : TWELVE_BYTES, __v : INTEGER, migrationId : STRING, dateMigrated : DATETIME) table was not created 
9279 [nioEventLoopGroup-3-4] DEBUG c.t.t.db.metaInf.CollectionMetaInfo - I will schedule creation of migrootions.(_id : TWELVE_BYTES, __v : INTEGER, migrationId : STRING, dateMigrated : DATETIME) table 
9279 [nioEventLoopGroup-3-4] DEBUG c.t.t.db.metaInf.CollectionMetaInfo - migrootions.(_id : TWELVE_BYTES, __v : INTEGER, migrationId : STRING, dateMigrated : DATETIME) table creation has been scheduled 
9364 [torod-system-1] DEBUG org.jooq.tools.LoggerListener - Affected row(s)          : 0 
9364 [torod-system-1] DEBUG org.jooq.tools.StopWatch - Query executed           : Total: 86.247ms 
9364 [torod-system-1] DEBUG org.jooq.tools.StopWatch - Finishing                : Total: 86.368ms, +0.121ms 
9368 [torod-system-1] DEBUG org.jooq.tools.LoggerListener - Executing query          : select "migrootions"."indexes"."index" from "migrootions"."indexes" 
9369 [torod-system-1] DEBUG org.jooq.tools.StopWatch - Query executed           : Total: 1.862ms 
9369 [torod-system-1] DEBUG org.jooq.tools.LoggerListener - Fetched result           : +-----+ 
9369 [torod-system-1] DEBUG org.jooq.tools.LoggerListener -                          : |index| 
9369 [torod-system-1] DEBUG org.jooq.tools.LoggerListener -                          : +-----+ 
9370 [torod-system-1] DEBUG org.jooq.tools.StopWatch - Finishing                : Total: 2.328ms, +0.466ms 
9395 [torod-system-1] DEBUG org.jooq.tools.LoggerListener - Calling routine          : { ? = call "public"."reserve_doc_ids"(?, ?) } 
9395 [torod-system-1] DEBUG org.jooq.tools.LoggerListener - -> with bind values      : { ? = call "public"."reserve_doc_ids"('migrootions', 50001) } 
9404 [torod-system-1] DEBUG org.jooq.tools.StopWatch - Query executed           : Total: 8.953ms 
9404 [torod-system-1] DEBUG org.jooq.tools.StopWatch - Fetching out values      : Total: 9.064ms, +0.111ms 
9405 [torod-system-1] DEBUG org.jooq.tools.LoggerListener - Fetched OUT parameters   : +------------+ 
9406 [torod-system-1] DEBUG org.jooq.tools.LoggerListener -                          : |RETURN_VALUE| 
9406 [torod-system-1] DEBUG org.jooq.tools.LoggerListener -                          : +------------+ 
9406 [torod-system-1] DEBUG org.jooq.tools.LoggerListener -                          : |       50000| 
9406 [torod-system-1] DEBUG org.jooq.tools.LoggerListener -                          : +------------+ 
9406 [torod-system-1] DEBUG org.jooq.tools.StopWatch - Out values fetched       : Total: 10.933ms, +1.869ms 
9406 [torod-system-1] DEBUG org.jooq.tools.StopWatch - Finishing                : Total: 11.031ms, +0.097ms 
9423 [torod-system-1] ERROR c.t.t.d.e.DefaultExecutorFactory - System executor exception 
com.torodb.torod.core.exceptions.ToroRuntimeException: java.lang.IllegalArgumentException: At the present time Torod doesn't support 'I' as identifier character. Only a alphanumeric letters and '_' are supported
        at com.torodb.torod.db.executor.jobs.SystemDbCallable.onFail(SystemDbCallable.java:64) ~[torodb.jar:na]
        at com.torodb.torod.db.executor.jobs.Job.call(Job.java:22) ~[torodb.jar:na]
        at com.torodb.torod.db.executor.DefaultSystemExecutor$SystemRunnable.call(DefaultSystemExecutor.java:153) ~[torodb.jar:na]
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.7.0_25]
        at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.7.0_25]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_25]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_25]
        at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
Caused by: java.lang.IllegalArgumentException: At the present time Torod doesn't support 'I' as identifier character. Only a alphanumeric letters and '_' are supported
        at com.torodb.torod.db.postgresql.IdsFilter.filter(IdsFilter.java:42) ~[torodb.jar:na]
        at com.torodb.torod.db.postgresql.IdsFilter.escapeAttributeName(IdsFilter.java:35) ~[torodb.jar:na]
        at com.torodb.torod.db.postgresql.meta.tables.SubDocTable.toColumnName(SubDocTable.java:244) ~[torodb.jar:na]
        at com.torodb.torod.db.postgresql.meta.tables.SubDocTable.<init>(SubDocTable.java:109) ~[torodb.jar:na]
        at com.torodb.torod.db.postgresql.meta.tables.SubDocTable.<init>(SubDocTable.java:100) ~[torodb.jar:na]
        at com.torodb.torod.db.postgresql.meta.tables.SubDocTable.<init>(SubDocTable.java:96) ~[torodb.jar:na]
        at com.torodb.torod.db.postgresql.meta.CollectionSchema.prepareSubDocTable(CollectionSchema.java:230) ~[torodb.jar:na]
        at com.torodb.torod.db.sql.AbstractSqlDbConnection.createSubDocTypeTable(AbstractSqlDbConnection.java:190) ~[torodb.jar:na]
        at com.torodb.torod.db.executor.jobs.CreateSubDocTableCallable.call(CreateSubDocTableCallable.java:58) ~[torodb.jar:na]
        at com.torodb.torod.db.executor.jobs.CreateSubDocTableCallable.call(CreateSubDocTableCallable.java:34) ~[torodb.jar:na]
        at com.torodb.torod.db.executor.jobs.SystemDbCallable.failableCall(SystemDbCallable.java:72) ~[torodb.jar:na]
        at com.torodb.torod.db.executor.jobs.Job.call(Job.java:20) ~[torodb.jar:na]
        ... 6 common frames omitted
@gortiz
Copy link
Contributor

gortiz commented Jun 8, 2015

First of all, I would like to apologize for not answering before.

This is a double bug. First, we do support simple queries on the metacollection system.indexes or system.namespaces, but not writes because there are other documents. This metacollections are deprecated in MongoDB 3.0 and substituted by some new commands like listIndexes. But we understand there are a lot of applications in the ecosystem that are not going to be updated until MongoDB remove the deprecated way to do the things, so we will try to support inserts on this metacollections as soon as we can.

The second bug you discovered is that we do not support collections with non alphanumeric or _ characters. This is an artificial restriction that we will remove very soon.

@gortiz
Copy link
Contributor

gortiz commented Jun 11, 2015

Commit 3e3838a implements the insert action on system.indexes (it is not supported on other metacollections!). It hasn't been tested as much as I would, so it is not going to be in master until next week.

@gortiz gortiz self-assigned this Jun 11, 2015
@okapies
Copy link
Author

okapies commented Jun 11, 2015

Thanks so much! I'll try it.

@ahachete
Copy link
Contributor

Hi okapies, did it work for you?

@okapies
Copy link
Author

okapies commented Jun 23, 2015

Oh, sorry. I don't try yet because I'm little bit busy now. :( Do you want reporting for it very soon?

@okapies
Copy link
Author

okapies commented Jun 27, 2015

Everything works fine!

> lets-chat@0.4.0 prestart /Users/okapy/workspace/lets-chat
> migroose


> lets-chat@0.4.0 start /Users/okapy/workspace/lets-chat
> node app.js


██╗     ███████╗████████╗███████╗     ██████╗██╗  ██╗ █████╗ ████████╗
██║     ██╔════╝╚══██╔══╝██╔════╝    ██╔════╝██║  ██║██╔══██╗╚══██╔══╝
██║     █████╗     ██║   ███████╗    ██║     ███████║███████║   ██║   
██║     ██╔══╝     ██║   ╚════██║    ██║     ██╔══██║██╔══██║   ██║   
███████╗███████╗   ██║   ███████║    ╚██████╗██║  ██║██║  ██║   ██║   
╚══════╝╚══════╝   ╚═╝   ╚══════╝     ╚═════╝╚═╝  ╚═╝╚═╝  ╚═╝   ╚═╝

Release 0.4.0

But I got another error at first access to the Let's Chat. I'll post that issue.

@okapies okapies mentioned this issue Jun 27, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gortiz gortiz

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@okapies @ahachete @gortiz