fix #11663, #11664, #11877: SQL DOMAIN; import logs (rebased); FS delete log trigger #2191

mtbc · 2014-03-21T12:45:15Z

Fixes:

Check that everything still seems to work: can import, view, delete images; can view import logs.

Breaking so needs label from @sbesson? Also @qidane should glance at some of the changes.

omitting x- prefix for http://tools.ietf.org/html/bcp178

to repeat in OMERO5.0__1 for those already on OMERO5.0__0

Conflicts: components/blitz/src/ome/formats/importer/ImportLibrary.java

sbesson · 2014-03-21T12:45:58Z

Breaking label added.

joshmoore · 2014-03-21T13:25:58Z

components/dsl/src/ome/dsl/Property.java

-            if (!getNullable()) {
-                sb.append(" not null");
-            }
-            if (getUnique()) {


Where are these UNIQUE and NOT NULL handled?

I too had wondered how they still take effect. I believe it is through getUnique() and getNullable() being accessed at http://github.com/openmicroscopy/openmicroscopy/blob/develop/components/dsl/resources/ome/dsl/object.vm#L1006.

I had removed them partly because, with them included, schema.sql gets a number of not null not null (sic) definitions.

mtbc · 2014-03-21T17:05:40Z

There is pleasing greenness at http://ci.openmicroscopy.org/view/Breaking/.

will-moore · 2014-03-24T14:18:18Z

Import, viewing and deleting images works fine. Noticed that import logs when viewed in Insight all terminate with "Step 4" as the last line, but this is also true in the 'latest' build on trout, so it's nothing to do with this PR.
All working fine.

mtbc · 2014-03-24T20:15:10Z

Well, the latest run of http://ci.openmicroscopy.org/view/Breaking/job/OMERO-5.1-breaking-integration-python/lastCompletedBuild/testReport/ failed but the same test failed in http://ci.openmicroscopy.org/view/5.1/job/OMERO-5.1-merge-integration-python/lastCompletedBuild/testReport/ without this PR: the usual "No valid permissions available!" gremlin.

joshmoore · 2014-03-25T01:49:48Z

Asked offline by @sbesson, I don't have any objections to this. I'm slightly worried about the drop of the UNIQUE and NOT NULL, but those should become apparent if the feature is every used. I also wonder if there is any issue with older PG versions. But as long as we have a ticket to test 5.1 with the oldest version of PG that we support (which we can discuss), then 👍 to have this in the merge build.

mtbc · 2014-03-25T08:41:33Z

In the conditional it was applying only for floats anyway; there is also, say, column share.data which is bytea not null. Once this is in I could open another, non-breaking, PR if you like that adds them in to columnDefinition for all types where applicable: I guess even if not needed they do no harm.

mtbc · 2014-03-25T09:06:52Z

In general we should have some testing (maybe a CI node) with minimum versions of prerequisites installed, not just PostgreSQL.

qidane · 2014-03-25T11:31:00Z

Looks good!

sbesson · 2014-03-25T11:51:25Z

breaking label removed

sbesson · 2014-03-25T12:00:38Z

[hudson@ome-c6100-3 ~]$ pg_dump -U omero -Fc -f ~/OMERO-5.1-merge-deploy-25032014.db.dump OMERO-5.1-merge-deploy
[hudson@ome-c6100-3 ~]$ file ~/OMERO-5.1-merge-deploy-25032014.db.dump
/home/hudson/OMERO-5.1-merge-deploy-25032014.db.dump: PostgreSQL custom database dump - v1.12-0
[hudson@ome-c6100-3 ~]$  psql -U omero OMERO-5.1-merge-deploy --set ON_ERROR_STOP=1 < /opt/hudson/workspace/OMERO-5.1-breaking-upgrade/VERSION/LAST_DEV/label/ome-c6100-3/OMERO.server/sql/psql/OMERO5.1DEV__2/OMERO5.1DEV__1.sql 
BEGIN
CREATE FUNCTION
 omero_assert_db_version 
-------------------------

(1 row)

DROP FUNCTION
INSERT 0 1
CREATE FUNCTION
ERROR:  update or delete on table "annotation" violates foreign key constraint "fkimageannotationlink_child_annotation" on table "imageannotationlink"
DETAIL:  Key (id)=(11305) is still referenced from table "imageannotationlink".
CONTEXT:  SQL statement "DELETE FROM annotation WHERE id = import.annotation_id"
PL/pgSQL function upgrade_import_logs() line 39 at SQL statement
[hudson@ome-c6100-3 ~]$

Need some investigation. Relabelled as breaking

mtbc · 2014-03-25T12:28:17Z

Somebody had actually attached import logs to an image! The upgrade script is now fixed to anticipate such strangeness.

mtbc · 2014-03-25T13:44:22Z

--rebased-to #2203 upgrade script change in latest commit

sbesson · 2014-03-25T16:38:28Z

[hudson@ome-c6100-3 ~]$ psql -U omero OMERO-5.1-merge-deploy --set ON_ERROR_STOP=1 < /opt/hudson/workspace/OMERO-5.1-breaking-upgrade/VERSION/LAST_DEV/label/ome-c6100-3/OMERO.server/sql/psql/OMERO5.1DEV__2/OMERO5.1DEV__1.sql 
BEGIN
CREATE FUNCTION
omero_assert_db_version 
-------------------------

(1 row)

DROP FUNCTION
INSERT 0 1
CREATE FUNCTION
upgrade_import_logs 
---------------------

(1 row)

DROP FUNCTION
CREATE FUNCTION
CREATE DOMAIN
CREATE DOMAIN
CREATE DOMAIN
CREATE DOMAIN
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ERROR:  value for domain positive_float violates check constraint "positive_float_check"

As discussed in devteam, this is related to some kymograph images with a zero p.physicalsizey value. Another edge-case to handle as part of this script (nullify the illegal values as suggested by @mtbc?)

Additionally, we should fix our kymograph script to prevent zero-valued physical sizes to be set /cc @will-moore

mtbc · 2014-03-25T16:46:58Z

Nulling done in latest commit and http://trac.openmicroscopy.org.uk/ome/ticket/12128 filed about kymograph script.

sbesson · 2014-03-27T07:28:04Z

[hudson@ome-c6100-3 ~]$  pg_dump -U omero -Fc -f ~/OMERO-5.1-merge-deploy-27032014.db.dump OMERO-5.1-merge-deploy
[hudson@ome-c6100-3 ~]$ file ~/OMERO-5.1-merge-deploy-25032014.db.dump
/home/hudson/OMERO-5.1-merge-deploy-25032014.db.dump: PostgreSQL custom database dump - v1.12-0
[hudson@ome-c6100-3 ~]$ psql -U omero OMERO-5.1-merge-deploy --set ON_ERROR_STOP=1 < /opt/hudson/workspace/OMERO-5.1-breaking-upgrade/VERSION/LAST_DEV/label/ome-c6100-3/OMERO.server/sql/psql/OMERO5.1DEV__2/OMERO5.1DEV__1.sql 
BEGIN
CREATE FUNCTION
 omero_assert_db_version 
-------------------------

(1 row)

DROP FUNCTION
INSERT 0 1
CREATE FUNCTION
 upgrade_import_logs 
---------------------

(1 row)

DROP FUNCTION
CREATE FUNCTION
CREATE DOMAIN
CREATE DOMAIN
CREATE DOMAIN
CREATE DOMAIN
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
UPDATE 0
UPDATE 3
UPDATE 0
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
UPDATE 1
                                 status                                 
------------------------------------------------------------------------
                                                                       +
                                                                       +
                                                                       +
 YOU HAVE SUCCESSFULLY UPGRADED YOUR DATABASE TO VERSION OMERO5.1DEV__2+
                                                                       +
                                                                       +

(1 row)

COMMIT
[hudson@ome-c6100-3 ~]$

exclude label removed and http://ci.openmicroscopy.org/job/OMERO-5.1-merge-daily/86/ restarted

ximenesuk · 2014-03-27T13:35:14Z

I've checked the DB upgrade; importing, viewing, deleting and image; viewing the import log; the relevant integration test and the relevant tables in the DB. Everything looks good and assuming everyone is otherwise happy with this PR having been in the breaking job up until now then I'd say this is good to merge.

sbesson · 2014-03-27T15:08:26Z

Waiting for @joshmoore and/or @jburel to sign-off on this PR. OMERO-5.1-latest-deploy will need to be ugpraded.

joshmoore · 2014-04-01T18:47:07Z

👍 for this. Leaving @sbesson to run the upgrade script.

fix #11663, #11664, #11877: SQL DOMAIN; import logs (rebased); FS delete log trigger

sbesson · 2014-04-01T18:47:20Z

[hudson@ome-c6100-3 ~]$ pg_dump -U omero -Fc -f ~/OMERO-5.1-latest-deploy-01042014.db.dump OMERO-5.1-latest-deploy
[hudson@ome-c6100-3 ~]$ file ~/OMERO-5.1-latest-deploy-01042014.db.dump
/home/hudson/OMERO-5.1-latest-deploy-01042014.db.dump: PostgreSQL custom database dump - v1.12-0
[hudson@ome-c6100-3 ~]$  psql -U omero OMERO-5.1-latest-deploy --set ON_ERROR_STOP=1 < /opt/hudson/workspace/OMERO-5.1-breaking-upgrade/VERSION/LAST_DEV/label/ome-c6100-3/OMERO.server/sql/psql/OMERO5.1DEV__2/OMERO5.1DEV__1.sql 
BEGIN
CREATE FUNCTION
 omero_assert_db_version 
-------------------------

(1 row)

DROP FUNCTION
INSERT 0 1
CREATE FUNCTION
 upgrade_import_logs 
---------------------

(1 row)

DROP FUNCTION
CREATE FUNCTION
CREATE DOMAIN
CREATE DOMAIN
CREATE DOMAIN
CREATE DOMAIN
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
UPDATE 0
UPDATE 0
UPDATE 0
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
UPDATE 1
                                 status                                 
------------------------------------------------------------------------
                                                                       +
                                                                       +
                                                                       +
 YOU HAVE SUCCESSFULLY UPGRADED YOUR DATABASE TO VERSION OMERO5.1DEV__2+
                                                                       +
                                                                       +

(1 row)

COMMIT
[hudson@ome-c6100-3 ~]$

mtbc · 2014-04-01T19:24:33Z

At least it ran smoothly that time! Thank you all.

mtbc · 2014-04-04T11:32:03Z

--rebased-from #2055
(moved from description so scc can see all)

mtbc and others added 14 commits March 21, 2014 12:27

add media type for import logs

31f9f13

omitting x- prefix for http://tools.ietf.org/html/bcp178

adjust how the import log is created and linked to the fileset

f07d713

add upgrade script to fix old import logs

2f68973

to repeat in OMERO5.0__1 for those already on OMERO5.0__0

add import log find method to the metadata service

a45add0

adjust import library to use new log finding method

f610d7f

Conflicts: components/blitz/src/ome/formats/importer/ImportLibrary.java

add integration testing for new metadata service method

73479d9

Load log files using new method and not file annotation.

7e7e86d

Fix copyright and space.

e91d01e

permit empty lists of images to be queried for import logs

5d76087

include import log in fileset graph traversal

dd16972

add new SQL DOMAIN types

fc23258

adjust Java DSL code to use new SQL DOMAIN types

a586251

fix definition of sizes to match model

fdd2155

update patch number and upgrade scripts

8992478

mtbc changed the title ~~fix #11663, #11877: SQL DOMAIN; import logs (rebased); FS delete log trigger~~ fix #11663, #11664, #11877: SQL DOMAIN; import logs (rebased); FS delete log trigger Mar 21, 2014

sbesson added the breaking label Mar 21, 2014

sbesson added the develop label Mar 21, 2014

mtbc mentioned this pull request Mar 21, 2014

fix #11877: move import logs to upload jobs instead of annotations #2055

Merged

joshmoore reviewed Mar 21, 2014
View reviewed changes

adjust delete log trigger in generated schema

73d0441

sbesson removed the breaking label Mar 25, 2014

sbesson added the breaking label Mar 25, 2014

remove import logs as annotations on other entities

9c72ddc

mtbc mentioned this pull request Mar 25, 2014

remove import logs as annotations on other entities #2203

Merged

null physical sizes of pixels objects that are illegal in model

36d8ceb

sbesson removed the breaking label Mar 27, 2014

joshmoore added a commit that referenced this pull request Apr 1, 2014

Merge pull request #2191 from mtbc/model-changes

09cd2fb

fix #11663, #11664, #11877: SQL DOMAIN; import logs (rebased); FS delete log trigger

joshmoore merged commit 09cd2fb into ome:develop Apr 1, 2014

mtbc deleted the model-changes branch April 1, 2014 19:24

mtbc mentioned this pull request May 21, 2014

fix upgrade scripts for PostgreSQL 8.4 compatibility #2529

Merged

sbesson added this to the 5.1.0-m1 milestone Oct 14, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix #11663, #11664, #11877: SQL DOMAIN; import logs (rebased); FS delete log trigger #2191

fix #11663, #11664, #11877: SQL DOMAIN; import logs (rebased); FS delete log trigger #2191

mtbc commented Mar 21, 2014

sbesson commented Mar 21, 2014

joshmoore Mar 21, 2014

mtbc Mar 21, 2014

mtbc Mar 28, 2014

mtbc commented Mar 21, 2014

will-moore commented Mar 24, 2014

mtbc commented Mar 24, 2014

joshmoore commented Mar 25, 2014

mtbc commented Mar 25, 2014

mtbc commented Mar 25, 2014

qidane commented Mar 25, 2014

sbesson commented Mar 25, 2014

sbesson commented Mar 25, 2014

mtbc commented Mar 25, 2014

mtbc commented Mar 25, 2014

sbesson commented Mar 25, 2014

mtbc commented Mar 25, 2014

sbesson commented Mar 27, 2014

ximenesuk commented Mar 27, 2014

sbesson commented Mar 27, 2014

joshmoore commented Apr 1, 2014

sbesson commented Apr 1, 2014

mtbc commented Apr 1, 2014

mtbc commented Apr 4, 2014

fix #11663, #11664, #11877: SQL DOMAIN; import logs (rebased); FS delete log trigger #2191

fix #11663, #11664, #11877: SQL DOMAIN; import logs (rebased); FS delete log trigger #2191

Conversation

mtbc commented Mar 21, 2014

sbesson commented Mar 21, 2014

joshmoore Mar 21, 2014

Choose a reason for hiding this comment

mtbc Mar 21, 2014

Choose a reason for hiding this comment

mtbc Mar 28, 2014

Choose a reason for hiding this comment

mtbc commented Mar 21, 2014

will-moore commented Mar 24, 2014

mtbc commented Mar 24, 2014

joshmoore commented Mar 25, 2014

mtbc commented Mar 25, 2014

mtbc commented Mar 25, 2014

qidane commented Mar 25, 2014

sbesson commented Mar 25, 2014

sbesson commented Mar 25, 2014

mtbc commented Mar 25, 2014

mtbc commented Mar 25, 2014

sbesson commented Mar 25, 2014

mtbc commented Mar 25, 2014

sbesson commented Mar 27, 2014

ximenesuk commented Mar 27, 2014

sbesson commented Mar 27, 2014

joshmoore commented Apr 1, 2014

sbesson commented Apr 1, 2014

mtbc commented Apr 1, 2014

mtbc commented Apr 4, 2014