Major upgrade of TaQL

[gervandiepen]

TaQL has been extended with several new features:

• support of masked array making something like mean(DATA[FLAG]) possible
• an array column to be updated can be masked (a la numpy)
• better handling of units in divisions
• addition of ALTER TABLE command to add/remove/rename columns and keywords
• several new functions like diagonal and type coercion functions
• more control how to create an output table
• nr of rows can be given in CREATE TABLE command, which can be used as nested query
• addition of SHOW/HELP command to show help info, table structure, units, etc.
• optional concatenation of subtables when table concatenation is used
• many other small additions and bug fixes

[tammojan]

    Start 247: tTableGramCretab

247: Test command: /Users/dijkema/opt/casacore/src/cmake/cmake_assay "./tTableGramCretab"
247: Test timeout computed to be: 1500
247: 27c27
247: <   Stored as MultiFile with blocksize 4194304
247: ---
247: >   Stored as MultiFile with blocksize 32768
247: 81c81
247: <   Stored as MultiFile with blocksize 4194304
247: ---
247: >   Stored as MultiFile with blocksize 32768
247: FAIL (floating point discrepancies > 1e-5): ./tTableGramCretab
1/1 Test #247: tTableGramCretab .................***Failed    0.40 sec

[tammojan]

Thanks for assigning it. I have tested this branch previously offline. I think we need more feedback before merging this large commit.

Could someone run the Casa test suite of this branch?

Any thoughts @juliantaylor?

[juliantaylor]

the casa testsuite fails with this change:

two parse errors:

RuntimeError: Error in TaQL command: SELECT FROM /data/ramdisk/users/buildacc/jenkins/src-test/test-test_slsearch/nosedir/biglist.tbl WHERE (FREQUENCY BETWEEN 0 AND 100) AND (RECOMMENDED) AND (CHEMICAL_NAME IN ('Silicon Monocarbide', 'Potassium chloride')) AND ((LINELIST = 'Recomb') OR ((LINELIST != 'Recomb'))) ORDER BY FREQUENCY
  parse error at or near position 196 ','

RuntimeError: Error in TaQL command: SELECT FROM /data/ramdisk/users/buildacc/jenkins/src-test/test-test_slsearch/nosedir/biglist.tbl WHERE (FREQUENCY BETWEEN 0 AND 100) AND (SPECIES IN ('S18O', 'HC5Nv11=1')) AND (RECOMMENDED) AND ((LINELIST = 'Recomb') OR ((LINELIST != 'Recomb'))) ORDER BY FREQUENCY
  parse error at or near position 157 ','

and one other failure:

image::getprofile (file /data/compressed/users/jenkinsBuild/workspace/user-casa-build-el6-branch/gcwrap/tools/images/image_cmpt.cc, line 2241)  Exception Reported: Exception: Specified axis 2 is not a direction axis but has length > 1. Cannot do flux density calculation.

[tammojan]

The parse error is a nice catch.

Current trunk works:

TaQL> select APPLICATION from ~/leah.MS::HISTORY where APPLICATION in ('NDPPP')
using style python select APPLICATION from ~/leah.MS::HISTORY where APPLICATION in ('NDPPP')
    has been executed
    select result of 1 rows
1 selected columns:  APPLICATION
NDPPP

Whereas the branch fails:

TaQL> select APPLICATION from ~/projects/tim/tim.MS::HISTORY where APPLICATION in ('BBS', 'BBS')
Error in TaQL command: using style python select APPLICATION from ~/projects/tim/tim.MS::HISTORY where APPLICATION in ('BBS', 'BBS')
  parse error at or near position 102 ','

[tammojan]

Setting to 'dontmerge' to prevent accidentally merging this. At least we should look into the casa test failures.

[tammojan]

I have a build of this branch running in a web interface (to be announced when it's more finished). But it might be useful for having a look at some of the new features. http://taql.astron.nl

[gervandiepen]

I had to remove the possibility to enclose a set in parentheses, because it
caused a shift-reduce error. Could it be enclosed in square brackets in the
CASA code? This change is fully backward compatible.

I do not understand the 2nd problem. It seems like a coordinate problem,
which I haven't touched. Maybe the problem is in the to/fromRecord
function. Has CASA been built with the commits of last week?

On Mon, Apr 4, 2016 at 2:44 PM, Julian Taylor notifications@github.com
wrote:

the casa testsuite fails with this change:

two parse errors:

RuntimeError: Error in TaQL command: SELECT FROM /data/ramdisk/users/buildacc/jenkins/src-test/test-test_slsearch/nosedir/biglist.tbl WHERE (FREQUENCY BETWEEN 0 AND 100) AND (RECOMMENDED) AND (CHEMICAL_NAME IN ('Silicon Monocarbide', 'Potassium chloride')) AND ((LINELIST = 'Recomb') OR ((LINELIST != 'Recomb'))) ORDER BY FREQUENCY
parse error at or near position 196 ','

RuntimeError: Error in TaQL command: SELECT FROM /data/ramdisk/users/buildacc/jenkins/src-test/test-test_slsearch/nosedir/biglist.tbl WHERE (FREQUENCY BETWEEN 0 AND 100) AND (SPECIES IN ('S18O', 'HC5Nv11=1')) AND (RECOMMENDED) AND ((LINELIST = 'Recomb') OR ((LINELIST != 'Recomb'))) ORDER BY FREQUENCY
parse error at or near position 157 ','

and one other failure:

image::getprofile (file /data/compressed/users/jenkinsBuild/workspace/user-casa-build-el6-branch/gcwrap/tools/images/image_cmpt.cc, line 2241) Exception Reported: Exception: Specified axis 2 is not a direction axis but has length > 1. Cannot do flux density calculation.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#388 (comment)

[gervandiepen]

I've looked at the parse error and I must confess I was a bit sloppy.
TaQL always had 1 shift/reduce error caused by the parentheses around a
set. I've introduced another one and thought I had solved the problem by
removing the set parentheses. I'll fix it, so you can leave the CASA code
untouched.
But as I said, the 2nd image_cmpt problem must be something else, which
requires investigation which (I'm afraid) the CASA group has to do.

On Mon, Apr 4, 2016 at 3:19 PM, Ger van Diepen gervandiepen@gmail.com
wrote:

I had to remove the possibility to enclose a set in parentheses, because
it caused a shift-reduce error. Could it be enclosed in square brackets in
the CASA code? This change is fully backward compatible.

I do not understand the 2nd problem. It seems like a coordinate problem,
which I haven't touched. Maybe the problem is in the to/fromRecord
function. Has CASA been built with the commits of last week?

On Mon, Apr 4, 2016 at 2:44 PM, Julian Taylor notifications@github.com
wrote:

the casa testsuite fails with this change:

two parse errors:

RuntimeError: Error in TaQL command: SELECT FROM /data/ramdisk/users/buildacc/jenkins/src-test/test-test_slsearch/nosedir/biglist.tbl WHERE (FREQUENCY BETWEEN 0 AND 100) AND (RECOMMENDED) AND (CHEMICAL_NAME IN ('Silicon Monocarbide', 'Potassium chloride')) AND ((LINELIST = 'Recomb') OR ((LINELIST != 'Recomb'))) ORDER BY FREQUENCY
parse error at or near position 196 ','

RuntimeError: Error in TaQL command: SELECT FROM /data/ramdisk/users/buildacc/jenkins/src-test/test-test_slsearch/nosedir/biglist.tbl WHERE (FREQUENCY BETWEEN 0 AND 100) AND (SPECIES IN ('S18O', 'HC5Nv11=1')) AND (RECOMMENDED) AND ((LINELIST = 'Recomb') OR ((LINELIST != 'Recomb'))) ORDER BY FREQUENCY
parse error at or near position 157 ','

and one other failure:

image::getprofile (file /data/compressed/users/jenkinsBuild/workspace/user-casa-build-el6-branch/gcwrap/tools/images/image_cmpt.cc, line 2241) Exception Reported: Exception: Specified axis 2 is not a direction axis but has length > 1. Cannot do flux density calculation.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#388 (comment)

[tammojan]

@juliantaylor , could you paste the exception-throwing lines of gcwrap/tools/images/image_cmpt.cc (around line 2241, in the version on your jenkins) here? Perhaps that will give Ger a clue on what the offending change was (and possibly how it can be fixed, in either casa or here).

[juliantaylor]

actually the line I pasted was not the test failure ... it was another test checking a mean which exceeded the tolerance. But that test uses random numbers and does not fail anymore, maybe it was just a random deviation.
But I noticed a bunch of numbers in a test have changed slightly, why does this change numerical results?

[juliantaylor]

I mean the changes to meas/MeasUDF/test/tmeas.out

[tammojan]

In e6d346a, @gervandiepen changed the input to another date, so that it does not depend on measures data newer than 2016. With changing the input date, also the output of the measures conversion changes.

[tammojan]

The input date is in meas/MeasUDF/test/tmeas.run

[juliantaylor]

ok good, the failure was probably just a fluke then. So only the set parenthesis issue remains.

[tammojan]

@gervandiepen It would be great if you could keep the in ("string1", "string2") syntax, since this is also standard SQL.

[gervandiepen]

I can keep it and I've already made the change in my working space.
I'll commit it today.
Isn't the reported image_cmpt.cc failure a problem anymore?

On Tue, Apr 5, 2016 at 4:04 PM, Tammo Jan Dijkema notifications@github.com
wrote:

@gervandiepen https://github.com/gervandiepen It would be great if you
could keep the in ("string1", "string2") syntax, since this is also
standard SQL.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#388 (comment)

[tammojan]

Isn't the reported image_cmpt.cc failure a problem anymore?

@juliantaylor said that was probably just a fluke. Not sure what that is, but it seems to have gone away.

[tammojan]

Checks pass now. Unless someone objects, I'll merge this tomorrow.