HPCC-11151 Support xpath(‘<>’) for write #10348

afishbeck · 2017-08-20T00:13:55Z

When #OPTION('writeInlineContent', true); is used writing a field with
XPATH('Name/<>') will output the string without XML/JSON encoding
the content. This allows users to insert arbitrary XML/JSON into
the output.

Also works with sets of strings XPATH('SetName/Name/<>').

There are 3 forms:
'<>' Writes the content without a root tag. Reading reads all
content of parent.
'Name<>' Writes the content without a tag. Reading reads the
content of the tag .
'Name/<>' Writes the content inside . Reading reads the
content of the tag .

Also fixes generated schemas to indicate locations where arbitrary
content might appear.

Signed-off-by: Anthony Fishbeck anthony.fishbeck@lexisnexis.com

Type of change:

This change is a bug fix (non-breaking change which fixes an issue).
This change is a new feature (non-breaking change which adds functionality).
This change improves the code (refactor or other change that does not change the functionality)
This change fixes warnings (the fix does not alter the functionality or the generated code)
This change is a breaking change (fix or feature that will cause existing behavior to change).
This change alters the query API (existing queries will have to be recompiled)

Checklist:

Testing:

When #OPTION('writeInlineContent', true); is used writing a field with XPATH('Name/<>') will output the string without XML/JSON encoding the content. This allows users to insert arbitrary XML/JSON into the output. Also works with sets of strings XPATH('SetName/Name/<>'). There are 3 forms: '<>' Writes the content without a root tag. Reading reads all content of parent. 'Name<>' Writes the content without a <Name> tag. Reading reads the content of the tag <Name>. 'Name/<>' Writes the content inside <Name>. Reading reads the content of the tag <Name>. Also fixes generated schemas to indicate locations where arbitrary content might appear. Signed-off-by: Anthony Fishbeck <anthony.fishbeck@lexisnexis.com>

HPCCSmoketest · 2017-08-20T00:50:36Z

Automated Smoketest: ✅
Sha: 66eeb6c
Build: success
Build: success
ECL Watch: Rebuilding Site

errors	warnings	build time
0	75	43.914 seconds

Install hpccsystems-platform-community_6.5.0-trunk0.el7.x86_64.rpm
HPCC Start: OK

Unit tests result:

Test	total	passed
unittest	91	91
wutoolTest(Dali)	19	19
wutoolTest(Cassandra)	19	19

Regression test result:

phase	total	pass
setup (hthor)	11	11
setup (thor)	11	11
setup (roxie)	11	11
test (hthor)	746	746
test (thor)	641	641
test (roxie)	774	774

HPCC Stop: OK
HPCC Uninstall: OK

afishbeck · 2017-08-24T19:51:35Z

@richardkchapman you may want to take a look at where this overlapped with your dynamic typeinfo changes.
@ghalliday Can you review?

ghalliday

@afishbeck some initial comments. It is probably worth me reviewing it all again later because it is quite a lot to take in at once.

Main issues are utf8 encoding of strings, and main comment is using enumerations instead of byte.

ghalliday · 2017-09-04T11:35:07Z

common/deftype/deftype.cpp

+            s.append("><xs:annotation><xs:appinfo hpcc:keyed=\"true\"/></xs:annotation></xs:any>\n");
+        else
+            s.append("/>\n");
+


trivial: extra newline

ghalliday · 2017-09-04T11:38:22Z

common/deftype/deftype.hpp

@@ -302,8 +302,8 @@ typedef IArrayOf<ITypeInfo> TypeInfoArray;
 interface ISchemaBuilder
 {
 public:
-    virtual void addField(const char * name, ITypeInfo & type, bool keyed) = 0;
-    virtual void addSetField(const char * name, const char * itemname, ITypeInfo & type) = 0;
+    virtual void addField(const char * name, ITypeInfo & type, bool keyed, byte contentFlags) = 0;


style: Cleaner to pass a named enumeration instead of a byte.

ghalliday · 2017-09-04T11:44:12Z

ecl/hql/hqlutil.hpp

+{
+    XPathContentInline      = 0x01,
+    XPathContentNamed       = 0x02,
+    XPathContentMixed       = 0x04


I don't think this is used. Could you reuse the previous enumeration, rather than defining a new one?

ghalliday · 2017-09-04T11:48:27Z

common/fileview2/fvresultset.cpp

        break;
    case type_unicode:
        len = getLength(type, cur);
        writer.outputUnicode(len, (UChar const *)cur, name);
+        //tbd


MORE: Is this going to be implemented?

ghalliday · 2017-09-04T11:51:36Z

common/fileview2/fvresultset.cpp

@@ -1607,7 +1628,10 @@ void CResultSetCursor::writeXmlText(IXmlWriter &writer, int columnIndex, const c
    case type_qstring:
        len = getLength(type, cur);
        rtlQStrToStrX(resultLen, resultStr, len, (const char *)cur);
-        writer.outputString(resultLen, resultStr, name);
+        if (meta.allowInlineContent && (contentFlags & XSBLD_contentInline))
+            writer.outputInline(resultLen, resultStr, (contentFlags & XSBLD_contentNamed) ? name : nullptr);


Although this is output inline it will still need to be converted from iso-8859-1 codepage to utf8.

To test ensure the strings contain some latin accented characters.

ghalliday · 2017-09-04T12:07:58Z

ecl/hqlcpp/hqlhtcpp.cpp

@@ -10923,7 +10930,7 @@ void HqlCppTranslator::buildXmlSerializeSetValues(BuildCtx & ctx, IHqlExpression
        CHqlBoundExpr boundCurElement;
        cursor->buildIterateLoop(loopctx, boundCurElement, false);
        OwnedHqlExpr curElement = boundCurElement.getTranslatedExpr();
-        buildXmlSerializeScalar(loopctx, curElement, itemName);
+        buildXmlSerializeScalar(loopctx, curElement, itemName, false);


The fourth parameter is a contentFlag, not a boolean. Using an enum for the parameter would have caught it.

ghalliday · 2017-09-04T12:10:49Z

common/thorhelper/thorxmlwrite.cpp

@@ -117,6 +117,26 @@ void CommonXmlWriter::outputString(unsigned len, const char *field, const char *
    }
 }

+void CommonXmlWriter::outputInline(unsigned len, const char *field, const char *fieldname)


minor style: Cleaner if length parameter are size32_t rather than unsigned.

ghalliday · 2017-09-04T12:12:13Z

esp/services/ws_workunits/ws_workunitsService.cpp

@@ -3277,7 +3277,7 @@ bool CWsWorkunitsEx::onWUResult(IEspContext &context, IEspWUResultRequest &req,
        const char* resultName = req.getResultName();

        Owned<DataCacheElement> data;
-        if (!req.getBypassCachedResult())
+        if (0)//!req.getBypassCachedResult())


more: Is this left in by mistake?

ghalliday · 2017-09-04T12:12:58Z

rtl/eclrtl/rtlfield.cpp

@@ -679,11 +679,17 @@ size32_t RtlStringTypeInfo::toXML(const byte * self, const byte * selfrow, const
        unsigned lenAscii;
        rtlDataAttr ascii;
        rtlEStrToStrX(lenAscii, ascii.refstr(), thisLength, str);
-        target.outputString(lenAscii, ascii.getstr(), queryScalarXPath(field));
+        if (field->hasInlineContentXpath())
+            target.outputInline(lenAscii, ascii.getstr(), field->hasNamedContentXpath() ? queryScalarXPath(field) : nullptr);


Same issues of conversion to utf8.

ghalliday · 2017-09-04T12:15:52Z

rtl/eclrtl/rtlfield.cpp

-    target.outputUtf8(thisLength, str, queryScalarXPath(field));
+    if (field->hasInlineContentXpath())
+    {
+        target.outputInline(thisLength, str, field->hasNamedContentXpath() ? queryScalarXPath(field) : nullptr);


Need to be careful about length v size. (Worth naming size variables appropriately.)

richardkchapman · 2017-10-25T09:47:27Z

@afishbeck This seems to be stalled (and needs rebasing)

afishbeck · 2017-10-28T22:39:43Z

Will reopen when I return from PTO.

afishbeck force-pushed the writeInlineXmlz branch from fe6fce8 to 66eeb6c Compare August 20, 2017 00:21

ghalliday requested changes Sep 4, 2017

View reviewed changes

afishbeck closed this Oct 28, 2017

afishbeck deleted the writeInlineXmlz branch October 7, 2022 16:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HPCC-11151 Support xpath(‘<>’) for write #10348

HPCC-11151 Support xpath(‘<>’) for write #10348

afishbeck commented Aug 20, 2017 •

edited

HPCCSmoketest commented Aug 20, 2017

afishbeck commented Aug 24, 2017

ghalliday left a comment

ghalliday Sep 4, 2017

ghalliday Sep 4, 2017

ghalliday Sep 4, 2017

ghalliday Sep 4, 2017

ghalliday Sep 4, 2017

ghalliday Sep 4, 2017

ghalliday Sep 4, 2017

ghalliday Sep 4, 2017

ghalliday Sep 4, 2017

ghalliday Sep 4, 2017

richardkchapman commented Oct 25, 2017

afishbeck commented Oct 28, 2017

HPCC-11151 Support xpath(‘<>’) for write #10348

HPCC-11151 Support xpath(‘<>’) for write #10348

Conversation

afishbeck commented Aug 20, 2017 • edited

Type of change:

Checklist:

Testing:

HPCCSmoketest commented Aug 20, 2017

afishbeck commented Aug 24, 2017

ghalliday left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richardkchapman commented Oct 25, 2017

afishbeck commented Oct 28, 2017

afishbeck commented Aug 20, 2017 •

edited