New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scripts end prematurely and randomly when using loop #1106

Open
Galan2X opened this Issue Jun 2, 2016 · 27 comments

Comments

@Galan2X

Galan2X commented Jun 2, 2016

When running a script within Robomongo that includes document enumeration, such as:
db.MyCollection.find().forEach(function(doc) {
// do some work, or print some data
});

If the amount of work is not trivial (such as a 'printjson(doc)' only) and the collection has hundreds of documents, the enumeration ends at random times before the end is reached, as if the cursor is sometimes advanced more than one step, or it believes it has reached the end. There are no errors thrown. The same script run locally on the server with the MongoDB shell does not exhibit this behavior.

This is for Robomongo 0.9.0-RC8 on Mac. It did not occur on earlier versions of Robomongo, like probably pre-0.9.0.

@simsekgokhan

This comment has been minimized.

Show comment
Hide comment
@simsekgokhan

simsekgokhan Jun 6, 2016

Collaborator

Hi @Galan2X, we have reproduced the issue, we are now investigating the problem.
Thanks a lot for reporting.

Problem Summary:

Long running scripts (inserting) from shell ends before the end of script.

Investigation Details:

Steps to reproduce:
var document = {"link": {"status":65},"clients": {"enddt" : "test", "type" : "testtype", "clentid" : 1}};
for (var j=0; j<500; j++) // for j=100 working
{
db.aa6.insert(document)
}

Next Actions:

  • Investigate logs.
  • Check roboshell timeouts.
Collaborator

simsekgokhan commented Jun 6, 2016

Hi @Galan2X, we have reproduced the issue, we are now investigating the problem.
Thanks a lot for reporting.

Problem Summary:

Long running scripts (inserting) from shell ends before the end of script.

Investigation Details:

Steps to reproduce:
var document = {"link": {"status":65},"clients": {"enddt" : "test", "type" : "testtype", "clentid" : 1}};
for (var j=0; j<500; j++) // for j=100 working
{
db.aa6.insert(document)
}

Next Actions:

  • Investigate logs.
  • Check roboshell timeouts.

@juliashibalko juliashibalko changed the title from Scripts end prematurely and randomly to 0.9-RC8+ Scripts end prematurely and randomly when using loop Jun 29, 2016

@juliashibalko juliashibalko changed the title from 0.9-RC8+ Scripts end prematurely and randomly when using loop to Scripts end prematurely and randomly when using loop Jun 29, 2016

@juliashibalko juliashibalko added the bug label Jun 29, 2016

@sthammadi-spsc

This comment has been minimized.

Show comment
Hide comment
@sthammadi-spsc

sthammadi-spsc Jul 1, 2016

Just to add, this issue also happen on Windows OS.

Just to add, this issue also happen on Windows OS.

@simsekgokhan

This comment has been minimized.

Show comment
Hide comment
@simsekgokhan

simsekgokhan Jul 1, 2016

Collaborator

Thanks for the input Sandeep. We will investigate the issue as soon as possible.

Collaborator

simsekgokhan commented Jul 1, 2016

Thanks for the input Sandeep. We will investigate the issue as soon as possible.

@juliashibalko

This comment has been minimized.

Show comment
Hide comment
@juliashibalko

juliashibalko Jul 4, 2016

Collaborator

@Galan2X, @sthammadi-spsc, @oschrenk For Robomongo 0.9.x version as workaround let me suggest you next solution:

  1. Make sure Robomongo is closed

  2. Open robomongo.json configuration file .

[Update]: Adding Robomongo-Config-File-Guide

Please mind that for Robomongo 0.9.x version configuration file location differs from app 0.8.x version:

Windows
  0.9.x
  C:\Users\<user>\.config\robomongo\0.9\robomongo.json
  0.8.x
  C:\Users\<user>\.config\robomongo\robomongo.json   
MAC
    0.9.x
    /Users/<user>/.config/robomongo/0.9/robomongo.json
    0.8.x
   /Users/<user>/.config/robomongo/robomongo.json     
Linux
    0.9.x
    /home/<user>/.config/robomongo/0.9/robomongo.json
    0.8.x
    /home/<user>/.config/robomongo/robomongo.json
  1. Change "shellTimeoutSec" attribute value to higher number in seconds. (i.e. "shellTimeoutSec" : 60)
  2. Save config file and re-launch Robomongo app.
  3. Run necessary script.
    If script isn't executed entirely, try to increase shellTimeoutSec value.
    Please let me know in case of any questions
Collaborator

juliashibalko commented Jul 4, 2016

@Galan2X, @sthammadi-spsc, @oschrenk For Robomongo 0.9.x version as workaround let me suggest you next solution:

  1. Make sure Robomongo is closed

  2. Open robomongo.json configuration file .

[Update]: Adding Robomongo-Config-File-Guide

Please mind that for Robomongo 0.9.x version configuration file location differs from app 0.8.x version:

Windows
  0.9.x
  C:\Users\<user>\.config\robomongo\0.9\robomongo.json
  0.8.x
  C:\Users\<user>\.config\robomongo\robomongo.json   
MAC
    0.9.x
    /Users/<user>/.config/robomongo/0.9/robomongo.json
    0.8.x
   /Users/<user>/.config/robomongo/robomongo.json     
Linux
    0.9.x
    /home/<user>/.config/robomongo/0.9/robomongo.json
    0.8.x
    /home/<user>/.config/robomongo/robomongo.json
  1. Change "shellTimeoutSec" attribute value to higher number in seconds. (i.e. "shellTimeoutSec" : 60)
  2. Save config file and re-launch Robomongo app.
  3. Run necessary script.
    If script isn't executed entirely, try to increase shellTimeoutSec value.
    Please let me know in case of any questions
@sthammadi-spsc

This comment has been minimized.

Show comment
Hide comment
@sthammadi-spsc

sthammadi-spsc Jul 4, 2016

@Galan2X, thanks for the workaround. Unfortunately, I couldn't find 'shellTimeoutSec' attribute in my robomongo.json file. I'm assuming I could add it in.

Below is my config:
{
"autoExec" : true,
"autoExpand" : false,
"autocompletionMode" : 1,
"batchSize" : 50,
"connections" : [
// Removed other connections here
{
"connectionName" : "Localhost",
"credentials" : [
{
"databaseName" : "admin",
"enabled" : false,
"mechanism" : "SCRAM-SHA-1",
"userName" : "",
"userPassword" : ""
}
],
"defaultDatabase" : "",
"serverHost" : "localhost",
"serverPort" : 27017
}
],
"disableConnectionShortcuts" : false,
"lineNumbers" : false,
"loadMongoRcJs" : false,
"style" : "Native",
"textFontFamily" : "",
"textFontPointSize" : -1,
"timeZone" : 0,
"toolbars" : {
"connect" : true,
"exec" : true,
"explorer" : true,
"logs" : false,
"open_save" : true
},
"uuidEncoding" : 0,
"version" : "2.0",
"viewMode" : 1
}

@Galan2X, thanks for the workaround. Unfortunately, I couldn't find 'shellTimeoutSec' attribute in my robomongo.json file. I'm assuming I could add it in.

Below is my config:
{
"autoExec" : true,
"autoExpand" : false,
"autocompletionMode" : 1,
"batchSize" : 50,
"connections" : [
// Removed other connections here
{
"connectionName" : "Localhost",
"credentials" : [
{
"databaseName" : "admin",
"enabled" : false,
"mechanism" : "SCRAM-SHA-1",
"userName" : "",
"userPassword" : ""
}
],
"defaultDatabase" : "",
"serverHost" : "localhost",
"serverPort" : 27017
}
],
"disableConnectionShortcuts" : false,
"lineNumbers" : false,
"loadMongoRcJs" : false,
"style" : "Native",
"textFontFamily" : "",
"textFontPointSize" : -1,
"timeZone" : 0,
"toolbars" : {
"connect" : true,
"exec" : true,
"explorer" : true,
"logs" : false,
"open_save" : true
},
"uuidEncoding" : 0,
"version" : "2.0",
"viewMode" : 1
}

@juliashibalko

This comment has been minimized.

Show comment
Hide comment
@juliashibalko

juliashibalko Jul 5, 2016

Collaborator

@sthammadi-spsc For version 0.9.0-RC8 or 0.9.0-RC9 this parameter usually set by default.
Let me offer you just to double-check version of Robomongo using and configuration file location on C:\Users\<user>\.config\robomongo\0.9\robomongo.json.
If there is still no "shellTimeoutSec" parameter, you could add both missed attributes manually after "loadMongoRcJs" : false, line:

 "mongoTimeoutSec" : 10,
 "shellTimeoutSec" : 60,

Then save robomongo.json file and start Robomongo.
Please let me know if you have any questions.

Collaborator

juliashibalko commented Jul 5, 2016

@sthammadi-spsc For version 0.9.0-RC8 or 0.9.0-RC9 this parameter usually set by default.
Let me offer you just to double-check version of Robomongo using and configuration file location on C:\Users\<user>\.config\robomongo\0.9\robomongo.json.
If there is still no "shellTimeoutSec" parameter, you could add both missed attributes manually after "loadMongoRcJs" : false, line:

 "mongoTimeoutSec" : 10,
 "shellTimeoutSec" : 60,

Then save robomongo.json file and start Robomongo.
Please let me know if you have any questions.

@teliatko

This comment has been minimized.

Show comment
Hide comment
@teliatko

teliatko Jul 12, 2016

@juliashibalko hi, what does the mongoTimeoutSec mean? How is this timeout different from shellTimeoutSec? Where is which one used?

Btw. can you check my comment in #1098 (comment)

Thanks.

@juliashibalko hi, what does the mongoTimeoutSec mean? How is this timeout different from shellTimeoutSec? Where is which one used?

Btw. can you check my comment in #1098 (comment)

Thanks.

@juliashibalko

This comment has been minimized.

Show comment
Hide comment
@juliashibalko

juliashibalko Jul 12, 2016

Collaborator

Hi @teliatko,
Thanks for interest in Robomongo! As you know Robomongo embeds mongo shell. So we've set two different timeout value attributes for MongoDB shell and for Robomongo application itself. Therefore shellTimeoutSec is used for scripts running in the shell only (e.g. db.getCollection().find() executing) and mongoTimeoutSec is used for actions executed outside the shell (e.g. expanding collections list).
Hope I've answered your question!

Collaborator

juliashibalko commented Jul 12, 2016

Hi @teliatko,
Thanks for interest in Robomongo! As you know Robomongo embeds mongo shell. So we've set two different timeout value attributes for MongoDB shell and for Robomongo application itself. Therefore shellTimeoutSec is used for scripts running in the shell only (e.g. db.getCollection().find() executing) and mongoTimeoutSec is used for actions executed outside the shell (e.g. expanding collections list).
Hope I've answered your question!

@snarlynarwhal

This comment has been minimized.

Show comment
Hide comment
@snarlynarwhal

snarlynarwhal Sep 16, 2016

I just ran into a similar, if not the same problem. When attempting to iterate through a collection of 100,000 documents, it would stop about 1/4 of the way through. I ported my code to the C# driver and it worked like a charm.

I just ran into a similar, if not the same problem. When attempting to iterate through a collection of 100,000 documents, it would stop about 1/4 of the way through. I ported my code to the C# driver and it worked like a charm.

@ClarenceL

This comment has been minimized.

Show comment
Hide comment
@ClarenceL

ClarenceL Nov 23, 2016

Ya it happens to me when I'm running a long mongo script, usually iterating over a cursor and updating records, kind of annoying to have to sit there and re-execute the script multiple times.

It seems to terminate after a random amount of time too.

Ya it happens to me when I'm running a long mongo script, usually iterating over a cursor and updating records, kind of annoying to have to sit there and re-execute the script multiple times.

It seems to terminate after a random amount of time too.

@juliashibalko

This comment has been minimized.

Show comment
Hide comment
@juliashibalko

juliashibalko Nov 23, 2016

Collaborator

@ClarenceL thanks for reporting! Yes, there is an issue about random terminating (please check #1232). We are investigating the problem.

Collaborator

juliashibalko commented Nov 23, 2016

@ClarenceL thanks for reporting! Yes, there is an issue about random terminating (please check #1232). We are investigating the problem.

@jfamme

This comment has been minimized.

Show comment
Hide comment
@jfamme

jfamme Dec 5, 2016

So, upping my timeout values to 300 did the trick for me...I guess the ask for me would be: Obviously the application can properly complete my query, if given enough time...it would be nice if the app told you that it timed out, and maybe presented a way to increase that wait, without having to manually find and edit config files. Thanks!

jfamme commented Dec 5, 2016

So, upping my timeout values to 300 did the trick for me...I guess the ask for me would be: Obviously the application can properly complete my query, if given enough time...it would be nice if the app told you that it timed out, and maybe presented a way to increase that wait, without having to manually find and edit config files. Thanks!

@simsekgokhan

This comment has been minimized.

Show comment
Hide comment
@simsekgokhan

simsekgokhan Dec 8, 2016

Collaborator

Hi @jfamme, thanks a lot for suggestions. All make perfect sense.
We have created two issues for this problem. We will work on these when we will have chance.
#1254
#1253

Collaborator

simsekgokhan commented Dec 8, 2016

Hi @jfamme, thanks a lot for suggestions. All make perfect sense.
We have created two issues for this problem. We will work on these when we will have chance.
#1254
#1253

@snarlynarwhal

This comment has been minimized.

Show comment
Hide comment
@snarlynarwhal

snarlynarwhal Dec 9, 2016

This has been an issue for about 8 months.. and it's a pretty significant issue. :/ Can we get a status update? Has any progress been made?

snarlynarwhal commented Dec 9, 2016

This has been an issue for about 8 months.. and it's a pretty significant issue. :/ Can we get a status update? Has any progress been made?

@simsekgokhan

This comment has been minimized.

Show comment
Hide comment
@simsekgokhan

simsekgokhan Dec 12, 2016

Collaborator

Hi @snarlynarwhal , currently priority is delivering critical features like Replica Sets and Export/Import. And the number of contributors of Robomongo is not too many. This issue is tagged as high vote and major, for sure it will be one of the first to work after delivering features mentioned above.

The workaround here #1106 (comment) seems to be working for lots people. Can you let us know if this helps you for now? If you have another problem, we will be happy to help.

Collaborator

simsekgokhan commented Dec 12, 2016

Hi @snarlynarwhal , currently priority is delivering critical features like Replica Sets and Export/Import. And the number of contributors of Robomongo is not too many. This issue is tagged as high vote and major, for sure it will be one of the first to work after delivering features mentioned above.

The workaround here #1106 (comment) seems to be working for lots people. Can you let us know if this helps you for now? If you have another problem, we will be happy to help.

@snarlynarwhal

This comment has been minimized.

Show comment
Hide comment
@snarlynarwhal

snarlynarwhal Dec 12, 2016

Okay sounds good, thank you very much! :)

Okay sounds good, thank you very much! :)

@jeremycooper1

This comment has been minimized.

Show comment
Hide comment
@jeremycooper1

jeremycooper1 Dec 14, 2016

Hi I tried up shellTimeoutSec" : 120 - the query I'm trying to run in version 9 only takes 6 seconds, yet it still won't appear running:
db.currentOp().inprog.forEach( function(op) { if(op.secs_running > 5) printjson(op); } )
but does in version 8 - not sure this is the same issue of this thread

Hi I tried up shellTimeoutSec" : 120 - the query I'm trying to run in version 9 only takes 6 seconds, yet it still won't appear running:
db.currentOp().inprog.forEach( function(op) { if(op.secs_running > 5) printjson(op); } )
but does in version 8 - not sure this is the same issue of this thread

@juliashibalko

This comment has been minimized.

Show comment
Hide comment
@juliashibalko

juliashibalko Dec 16, 2016

Collaborator

@jeremycooper1 Yes, you are correct - it's another issue. So let's continue investigation in scope of #1259 card.

Collaborator

juliashibalko commented Dec 16, 2016

@jeremycooper1 Yes, you are correct - it's another issue. So let's continue investigation in scope of #1259 card.

@jsheely

This comment has been minimized.

Show comment
Hide comment
@jsheely

jsheely Feb 5, 2017

@simsekgokhan I'd consider that this is a higher priority issue than that of new features.

Given this example:
In its current form you run a script that loops over a collection of say 10,000 items and you insert them into another collection. The script says it ran successfully, no warnings, no errors.. But you only get 300 items in the new collection.

Now you're in a position where you could lose a significant amount of data. This is something I would urge you to re-consider the importance of if time permits.

The current work around still has some significant issues. Unless you set these timeouts to be in terms of hours it is still possible to get false positive of completion if your script would happen to run longer than the timeout.

jsheely commented Feb 5, 2017

@simsekgokhan I'd consider that this is a higher priority issue than that of new features.

Given this example:
In its current form you run a script that loops over a collection of say 10,000 items and you insert them into another collection. The script says it ran successfully, no warnings, no errors.. But you only get 300 items in the new collection.

Now you're in a position where you could lose a significant amount of data. This is something I would urge you to re-consider the importance of if time permits.

The current work around still has some significant issues. Unless you set these timeouts to be in terms of hours it is still possible to get false positive of completion if your script would happen to run longer than the timeout.

@simsekgokhan

This comment has been minimized.

Show comment
Hide comment
@simsekgokhan

simsekgokhan Feb 7, 2017

Collaborator

Hi @jsheely, I totally agree with you. Thanks for the example and reminding us.

Collaborator

simsekgokhan commented Feb 7, 2017

Hi @jsheely, I totally agree with you. Thanks for the example and reminding us.

@roberterdin

This comment has been minimized.

Show comment
Hide comment
@roberterdin

roberterdin Mar 8, 2017

I agree with @jsheely. The fact that there is no warning is very dangerous. I was just seconds away from a data loss that would have been very expensive to us. Data that was purchased from a third party was almost lost because two collections were supposedly merged and the obsolete collection deleted. Luckily I double-checked with a document at the end of the target collection so this was avoided.

I agree with @jsheely. The fact that there is no warning is very dangerous. I was just seconds away from a data loss that would have been very expensive to us. Data that was purchased from a third party was almost lost because two collections were supposedly merged and the obsolete collection deleted. Luckily I double-checked with a document at the end of the target collection so this was avoided.

@snarlynarwhal

This comment has been minimized.

Show comment
Hide comment
@snarlynarwhal

snarlynarwhal Mar 8, 2017

I think this needs to be resolved asap also.

I think this needs to be resolved asap also.

@gauravkalal

This comment has been minimized.

Show comment
Hide comment
@gauravkalal

gauravkalal Mar 16, 2017

@juliashibalko This solution works. Thank you.

gauravkalal commented Mar 16, 2017

@juliashibalko This solution works. Thank you.

@RokoMijic

This comment has been minimized.

Show comment
Hide comment
@RokoMijic

RokoMijic Apr 18, 2017

It might be a good idea to edit the error message that is shown to make the user aware of what happened and what the resolution is. I can't imagine that it would take long to edit the text of the error message to say something more sensible.

It might be a good idea to edit the error message that is shown to make the user aware of what happened and what the resolution is. I can't imagine that it would take long to edit the text of the error message to say something more sensible.

@simsekgokhan

This comment has been minimized.

Show comment
Hide comment
@simsekgokhan

simsekgokhan Apr 21, 2017

Collaborator

@RokoMijic , thanks a lot for your interest and suggestion. We will consider.

Collaborator

simsekgokhan commented Apr 21, 2017

@RokoMijic , thanks a lot for your interest and suggestion. We will consider.

simsekgokhan added a commit that referenced this issue May 19, 2017

#1106: Attempt to detect and inform user about popular problem where …
…the long lasting scripts were ending prematurely and silently due to shell timeout reached ("shellTimeoutSec" parameter in config file)

@simsekgokhan simsekgokhan self-assigned this May 19, 2017

@simsekgokhan simsekgokhan added this to Finished in Robomongo 1.1 May 19, 2017

simsekgokhan added a commit that referenced this issue May 20, 2017

#1106: Adding ChangeShellTimoeutDialog and functionality to be able t…
…o change shell timeout ("shellTimeoutSec") at program run time (and save into config file), effective immediately on all existing/new tabs and persistent after program restart.

@juliashibalko juliashibalko moved this from Fixed / Ready for testing to Done in Robomongo 1.1 Jun 12, 2017

@simsekgokhan

This comment has been minimized.

Show comment
Hide comment
@simsekgokhan

simsekgokhan Jun 22, 2017

Collaborator

Hoping the related code changes for this problem and UI option to configure shell timeout which are included in the latest version Robo 3t 1.1 will solve this problem.

More:
Blog section: Fix for Robo 3T Shell Timeout Issue
All sections: http://blog.robomongo.org/robomongo-is-robo-3t

Collaborator

simsekgokhan commented Jun 22, 2017

Hoping the related code changes for this problem and UI option to configure shell timeout which are included in the latest version Robo 3t 1.1 will solve this problem.

More:
Blog section: Fix for Robo 3T Shell Timeout Issue
All sections: http://blog.robomongo.org/robomongo-is-robo-3t

@RokoMijic

This comment has been minimized.

Show comment
Hide comment

@simsekgokhan Looks good!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment