Warning: Incorrect datetime value in Galaxy 16.01

[FredericBGA]

Hello

When I try to upload a file, I can see this error in the log file :

/softs/bioinfo/galaxy/.venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py:450: Warning: Incorrect datetime value: '2016-03-29 07:25:29.220Z' for column 'update_time' at row 32

On the interface, the upload job seems to be stuck. The history will never be refreshed.
Otherwise, if I manually click on the “refresh history” icon, it works. The error is not shown again and my file appears in the history.

I run a git pull (see git.txt)
git.txt

 * branch            HEAD       -> FETCH_HEAD
Updating 909a549..d15b103

Here is the log when I try to upload a file (in galaxylogFirst.txt file).
galaxylogFirst.txt

The history will never refresh automatically.
Otherwise, if I manually click on the “refresh history” icon, it works. The error is not shown again and my file appears in the history.

Here is the log after I click the “refresh history” icon.
galaxylogSecond.txt

Here is the result of uname -a

uname -a
Linux  2.6.32-573.8.1.el6.x86_64 #1 SMP Tue Nov 10 18:01:38 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

We run under PBS Pro but upload is run locally (config/job_conf.xml).
<tool id="upload1" destination="local"/>

Does anybody see this ? I hope it will be easy to debug because this is annoying.

Thank you.

Fred

[FredericBGA]

Hi,

Galaxy uses a mySQL database.
Is it possible that there's a problem of date contistency betwwen the database server and the galaxy server ?

Fred

[dannon]

Thanks for reporting this, your guess seems like it's on the right track. I'll try to set up a mysql database and reproduce this here today.

[nsoranzo]

@FredericBGA I've opened pull request #2042 which might fix this for you, if you have time to test it that would be appreciated.

[carlfeberhard]

@FredericBGA What version of MySQL are you using there?

[FredericBGA]

Hello

Server version: 5.5.21-log MySQL Community Server (GPL)

Fred

De : carlfeberhard [mailto:notifications@github.com]
Envoyé : mercredi 30 mars 2016 21:22
À : galaxyproject/galaxy galaxy@noreply.github.com
Cc : SAPET, Frederic frederic.sapet@biogemma.com
Objet : Re: [galaxyproject/galaxy] Warning: Incorrect datetime value in Galaxy 16.01 (#2037)

@FredericBGAhttps://github.com/FredericBGA What version of MySQL are you using there?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHubhttps://github.com//issues/2037#issuecomment-203593857

[FredericBGA]

Hi

Thank you for being so fast to work on this issue !

I’m not a git advanced user.
I’m still facing the same issue (/softs/bioinfo/galaxy-prod/.venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py:450: Warning: Incorrect datetime value: '2016-03-31 07:51:54.483Z' for column 'update_time' at row 36) after a git pull command.

git pull
remote: Counting objects: 4772, done.
remote: Compressing objects: 100% (32/32), done.
remote: Total 4772 (delta 3079), reused 3062 (delta 3062), pack-reused 1678
Receiving objects: 100% (4772/4772), 6.81 MiB | 3.67 MiB/s, done.
Resolving deltas: 100% (3862/3862), completed with 612 local objects.
From https://github.com/galaxyproject/galaxy

• branch HEAD -> FETCH_HEAD
  Updating d15b103..b160cd5

Do I need to launch a more specific git command in order to get your fix ?

Fred

De : Nicola Soranzo [mailto:notifications@github.com]
Envoyé : mercredi 30 mars 2016 20:25
À : galaxyproject/galaxy galaxy@noreply.github.com
Cc : SAPET, Frederic frederic.sapet@biogemma.com
Objet : Re: [galaxyproject/galaxy] Warning: Incorrect datetime value in Galaxy 16.01 (#2037)

@FredericBGAhttps://github.com/FredericBGA I've opened pull request #2042#2042 which might fix this for you, if you have time to test it that would be appreciated.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHubhttps://github.com//issues/2037#issuecomment-203567318

[nsoranzo]

@FredericBGA A git pull is not enough because the pull request has not been merged yet. Probably the easiest way to apply the changes is to download https://raw.githubusercontent.com/nsoranzo/galaxy/fix_for_2037/lib/galaxy/managers/base.py , e.g. with

wget https://raw.githubusercontent.com/nsoranzo/galaxy/fix_for_2037/lib/galaxy/managers/base.py

and then copy the download base.py in lib/galaxy/managers/ (overwriting the original one). Then restart Galaxy and test again.

[FredericBGA]

Hi

Ok, it’s done.

wget https://raw.githubusercontent.com/nsoranzo/galaxy/fix_for_2037/lib/galaxy/managers/base.py
mv base.py lib/galaxy/managers/base.py
rm lib/galaxy/managers/base.pyc
./run.sh restart

But the error is still here.

/softs/bioinfo/galaxy-prod/.venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py:450: Warning: Incorrect datetime value: '2016-03-31 11:50:33Z' for column 'update_time' at row 38
cursor.execute(statement, parameters)

it will be always thrown until I click the refresh icon.

Fred

De : Nicola Soranzo [mailto:notifications@github.com]
Envoyé : jeudi 31 mars 2016 12:29
À : galaxyproject/galaxy galaxy@noreply.github.com
Cc : SAPET, Frederic frederic.sapet@biogemma.com
Objet : Re: [galaxyproject/galaxy] Warning: Incorrect datetime value in Galaxy 16.01 (#2037)

@FredericBGAhttps://github.com/FredericBGA A git pull is not enough because the pull request has not been merged yet. Probably the easiest way to apply the changes is to download https://raw.githubusercontent.com/nsoranzo/galaxy/fix_for_2037/lib/galaxy/managers/base.py , e.g. with

wget https://raw.githubusercontent.com/nsoranzo/galaxy/fix_for_2037/lib/galaxy/managers/base.py

and then copy the download base.py in lib/galaxy/managers/ (overwriting the original one). Then restart Galaxy and test again.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHubhttps://github.com//issues/2037#issuecomment-203871173

[dannon]

@FredericBGA I'm not seeing this behavior on 16.01 but I am able to reproduce it in on dev. Are you certain you're actually on 16.01? What's the output of git status?

[FredericBGA]

Hi

git status

Not currently on any branch.

Changed but not updated:

(use "git add ..." to update what will be committed)

(use "git checkout -- ..." to discard changes in working directory)

modified: lib/galaxy/jobs/runners/init.py

modified: lib/galaxy/jobs/runners/drmaa.py

modified: lib/galaxy/jobs/runners/util/job_script/DEFAULT_JOB_FILE_TEMPLATE.sh

modified: lib/galaxy/managers/base.py

Untracked files:

(use "git add ..." to include in what will be committed)

.bash_history

.bashrc

.cache/

.drmaa/

.local/

.profile

.python-eggs/

.viminfo

GalaxyTest.e14122

GalaxyTest.e14205

GalaxyTest.o14122

GalaxyTest.o14205

NCPUS.txt

StartStopRestartGalaxy.sh

backup

bin/

databases

include/

lib/galaxy/tools/toolbox/filters/examples.py

lib/galaxy/tools/toolbox/filters/teams.py

lib/python2.7/

pip-selfcheck.json

projects

shed_tools/

static/images/LogoBGA-S-Trans.png

static/images/Universe.jpg

test.pbs.sh

tools/bga/

tools/data_source/echo.py

tools/data_source/echo.xml

tools/data_source/ucsc_archaea.xml

tools/data_source/ucsc_filter.py

tools/data_source/ucsc_proxy.py

tools/data_source/ucsc_proxy.xml

tools/data_source/ucsc_testproxy.xml

tools/ga/

tools/ggc/

tools/ggo/

tools/gmtd/

tools/others/

no changes added to commit (use "git add" and/or "git commit -a")

Fred

De : Dannon Baker [mailto:notifications@github.com]
Envoyé : jeudi 31 mars 2016 14:50
À : galaxyproject/galaxy galaxy@noreply.github.com
Cc : SAPET, Frederic frederic.sapet@biogemma.com
Objet : Re: [galaxyproject/galaxy] Warning: Incorrect datetime value in Galaxy 16.01 (#2037)

@FredericBGAhttps://github.com/FredericBGA I'm not seeing this behavior on 16.01 but I am able to reproduce it in on dev. Are you certain you're actually on 16.01? What's the output of git status?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHubhttps://github.com//issues/2037#issuecomment-203918255

[nsoranzo]

@dannon I'm pretty sure this happens only on dev.

[dannon]

@nsoranzo Yep, that's what I was seeing too.

Ok, @FredericBGA -- have you intentionally made any changes to the local repository? I'd recommend getting on a release branch and tracking that (or master), instead of getting changes from the dev branch like it looks like you are. Let me know if you need help with this.

[FredericBGA]

HI Dannon

Yes, i need help in order to be sure to get the latest code.
I thought that git pull was enough to do that :(

By the way, the only files I have modified myself are (in order to make galaxy running under PBSPRo) :
lib/galaxy/jobs/runners/init.py
lib/galaxy/jobs/runners/drmaa.py
lib/galaxy/jobs/runners/util/job_script/DEFAULT_JOB_FILE_TEMPLATE.sh

I’m really sorry if all of this in only related to a bad version.

Fred

De : Dannon Baker [mailto:notifications@github.com]
Envoyé : jeudi 31 mars 2016 15:09
À : galaxyproject/galaxy galaxy@noreply.github.com
Cc : SAPET, Frederic frederic.sapet@biogemma.com
Objet : Re: [galaxyproject/galaxy] Warning: Incorrect datetime value in Galaxy 16.01 (#2037)

@nsoranzohttps://github.com/nsoranzo Yep, that's what I was seeing too.

Ok, @FredericBGAhttps://github.com/FredericBGA -- have you intentionally made any changes to the local repository? I'd recommend getting on a release branch and tracking that (or master), instead of getting changes from the dev branch like it looks like you are. Let me know if you need help with this.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHubhttps://github.com//issues/2037#issuecomment-203929646

[dannon]

@FredericBGA No problem at all. First thing I'd do is save your changes. You can diff them out to a patch with 'git diff -p > mychanges.patch', and/or stash with 'git stash'.

Then you have two options, to check out 'master', which will be the latest stable release always, or checkout release_16.01 specifically.'

Little bit more info here: https://wiki.galaxyproject.org/News/2016_01_GalaxyRelease

But the relevant command to get on 'master', which will always point to the latest stable release, is:
git checkout master && pull --ff-only origin master

Once you've done that, you can either patch your changes back in, or try to unstash with git stash pop. That'll attempt to reapply, and should give you an informative message aobut whether or not that worked.

[FredericBGA]

Hi,

It seems that the checkout failed.

git checkout master && pull --ff-only origin master
Previous HEAD position was b160cd5... Merge pull request #2046 from martenson/jstree-dictobj
Switched to branch 'master'
-bash: pull: command not found

git checkout master && git pull --ff-only origin master
Already on 'master'

From https://github.com/galaxyproject/galaxy

• branch master -> FETCH_HEAD
  fatal: Not possible to fast-forward, aborting.

[FredericBGA]

Hi,

it seems to be ok with:

git checkout -b release_16.01 origin/release_16.01

Switched to a new branch 'release_16.01'

and then

>git pull
remote: Counting objects: 13, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 13 (delta 7), reused 7 (delta 7), pack-reused 2
Unpacking objects: 100% (13/13), done.
From https://github.com/galaxyproject/galaxy
 * branch            HEAD       -> FETCH_HEAD

[FredericBGA]

git branch
dev
master

• release_16.01

But the error is still here.

Fred

[carlfeberhard]

Can you reply with the results from git status | head -2? It should have a second line stating: Your branch is up-to-date with 'origin/release_16.01'. or similar.

[dannon]

Closing the issue as the problem in question is resolved as of #2062. @FredericBGA feel free to keep using the thread for us to sort out your deployment issue (or email galaxy-dev)

[FredericBGA]

Hello

git status
# On branch release_16.01
# Changed but not updated:

The error has gone.
But now, the history is not refreshed even more.
When I upload a file, I can see that an upload process is undergo, but "This job is waiting to run" is displayed.
When I launch a tool (ie: Select first lines from a dataset) the job is not displayed even more.

When I click on the refresh icon, everything is ok, the file is here and the result of the job too.

I have joined the log (with my comments in bracket)
galaxylogUpload.txt

Fred

[nsoranzo]

@carlfeberhard @dannon It seems that the Galaxy server is at time zone +2, while the database is using UTC.

[carlfeberhard]

I think we've got two issues in this thread: a) the update timezone bug and b) getting Fred's installation onto release_16.01 properly. I'll keep working on (a) elsewhere. If we can solve (b), he won't have to worry about (a).

@FredericBGA Ok, thanks for that info. I think you are still on the dev branch. Can you reply with the output of git branch -vv, please?

(And, to second Dannon's suggestion: we can move this to the 'galaxy-dev' mailing list (http://dev.list.galaxyproject.org/) or to irc (https://wiki.galaxyproject.org/Support/IRC), too)

[FredericBGA]

Hi,

I found the problem.
the ntpd daemon was not started :(
so my galaxy server had not the same date as others.

Fred

[FredericBGA]

Hi ,

>git branch -vv
  dev           183c3c1 Merge pull request #1279 from carlfeberhard/fix.casper
  master        8de7292 Biogemma Version
* release_16.01 ff48ad9 Merge pull request #2063 from dannon/dryrun

[nsoranzo]

@FredericBGA So this is working for you now?

[FredericBGA]

Hi all

Thank you very much for your time, I really appreciate your help.

upload works now, no more problem of refresh, it works with jobs too.

I'm not sure if my version is the good one but I agree to close the discussion here and to continue on galaxy-dev.

Fred