New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for external instances #30

Closed
MartijnR opened this Issue Feb 27, 2015 · 22 comments

Comments

Projects
None yet
4 participants
@MartijnR
Copy link
Contributor

MartijnR commented Feb 27, 2015

earlier discussion on ODK dev forum

  • Support for both select_one_from_file [FILENAME] and select_multiple_from_file [FILENAME]
  • The above types always create regular <select> or <select1> with a regular <itemset> and always add a single <instance id="" src="" /> to the model (unless that instance with that id already exists).
  • Filenames can have either .csv or .xml extensions.
  • instance id attribute = lowercased filename minus the extension: countries.csv => <instance id="countries">
  • instance src attribute for XML sources gets jr://file/ prefix and for CSV sources gets jr://file-csv/ prefix (similar to jr://image, jr://video and jr://audio).
  • the label ref of an itemset is always ref="label" because itext references couldn't work.

Examples

1a. XLSForm input (CSV file, no choice filter):

type name label choice_filter
select_one_from_file countries.csv country Pick one country

1b. XForm output:

Very similar to a regular select_one with choice filter (but in this case the choice filter is empty)

<model>
    <instance>
        ....
    </instance>
    <instance id="cities" src="jr://file-csv/cities.csv" />
</model>
<h:body>
    <select1 ref="/myform/country">
        <label>Country</label>
        <itemset nodeset="instance('countries')/root/item">
            <value ref="name"/>
            <label ref="label"/>
        </itemset>
    </select1>
</h:body>

2a. XLSForm input (XML file, no choice filter):

type name label choice_filter
select_one_from_file countries.xml country Pick one country

2b. XForm output:

Exactly the same syntax as used by CommCare for their many XML external instances

<model>
    <instance>
        ....
    </instance>
    <instance id="cities" src="jr://file/cities.xml" />
</model>
<h:body>
    <select1 ref="/myform/country">
        <label>Country</label>
        <itemset nodeset="instance('countries')/root/item">
            <value ref="name"/>
            <label ref="label"/>
        </itemset>
    </select1>
</h:body>

3a. XLSForm input (CSV file, with choice filter):

type name label choice_filter
select_multiple_from_file cities.csv city Pick some cities country = ${country}

3b. XForm output:

<model>
    <instance>
        ....
    </instance>
    <instance id="cities" src="jr://file-csv/cities.csv" />
</model>
<h:body>
    <select ref="/myform/country">
        <label>Country</label>
        <itemset nodeset="instance('cities')/root/item[country= /myform/country ]">
            <value ref="name"/>
            <label ref="label"/>
        </itemset>
    </select>
</h:body>

Notes:

  • this feature is not compatible with ODK Collect (and vice versa select_external and search are not compatible with Enketo)
  • the external data referred to in an XLSForm needs to always have a label column (that's an XLSForm restriction)
  • csv column headers may not contain a colon (this not relevant to Pyxform but to the xlsform.org documentation)
  • these XForms do not pass ODK Validate validation (I suspect it checks whether itemset nodeset paths exist), so to use this feature ODK Validate would have to be switched off or tweaked so it bypasses checks for paths that refer to external instances (i.e. that refer to instance(ID)/path/to/node when the targeted instance has a src attribute)
@ukanga

This comment has been minimized.

Copy link
Contributor

ukanga commented Mar 1, 2015

Any chance this will be added to javarosa/odk validate/odk collect family?

@MartijnR

This comment has been minimized.

Copy link
Contributor

MartijnR commented Mar 1, 2015

Hopefully. Otherwise we need to go ahead with building a new validator based around's enketo's XPath evaluator. I know Dimagi is using external XML instances a lot already (same as above). Not sure what validator they use.

The feature has now been added to Enketo Express.

@mberg

This comment has been minimized.

Copy link

mberg commented Mar 2, 2015

Martijn are you following the approach dimagi is using? Do they have their
own version of javarosa to support this?
On Mar 2, 2015 1:58 AM, "Martijn van de Rijdt" notifications@github.com
wrote:

Hopefully. Otherwise we need to go ahead with building a new validator
based around's enketo's XPath evaluator
enketo/enketo-validate#1. I know Dimagi is
using external XML instances a lot already (same as above). Not sure what
validator they use.

The feature is has now been added to Enketo Express.


Reply to this email directly or view it on GitHub
#30 (comment).

@MartijnR

This comment has been minimized.

Copy link
Contributor

MartijnR commented Mar 2, 2015

Yes, same approach. No re-inventing of any wheels. I just added CSV support in addition to XML. Dimagi has their own version of Javarosa that supports XML external instances as above.

@dorey

This comment has been minimized.

Copy link
Contributor

dorey commented Mar 18, 2015

👍 for referencing the URI of the external itemset from within the XForm.

@MartijnR

This comment has been minimized.

Copy link
Contributor

MartijnR commented Mar 18, 2015

This opens the way for some exciting future possibilities, because the URL published in the /xFormsManifest could be a dynamic document. E.g. a dynamically created XML 👍 (or CSV 👎 ) Document.... It could be external to KoBo/Ona too.

You could e.g. create a service that dynamically creates a document based on another survey or based on earlier responses to the same survey.

@mberg

This comment has been minimized.

Copy link

mberg commented Mar 19, 2015

Is this working?

On Wednesday, March 18, 2015, Martijn van de Rijdt notifications@github.com
wrote:

This opens the way for some exciting future possibilities, because the URL
posted in the /xFormsManifest could be a dynamic document. E.g. a
dynamically created XML (or CSV [image: 👎] ) Document.... It could be
external to KoBo/Ona too.

You could e.g. create a service that dynamically creates a document based
on another survey or earlier responses to the same survey.


Reply to this email directly or view it on GitHub
#30 (comment).

@MartijnR

This comment has been minimized.

Copy link
Contributor

MartijnR commented Mar 19, 2015

In Enketo Express it is working, yes.

@mberg

This comment has been minimized.

Copy link

mberg commented Mar 19, 2015

Awesome

On Thursday, March 19, 2015, Martijn van de Rijdt notifications@github.com
wrote:

In Enketo Express it is working, yes.


Reply to this email directly or view it on GitHub
#30 (comment).

@MartijnR

This comment has been minimized.

Copy link
Contributor

MartijnR commented Apr 9, 2015

One way around ODK Validate is to add dummy content to the external instance. Instead of:

<instance id="cities" src="jr://file/cities.xml" />

we do:

<instance id="cities" src="jr://file/cities.xml" >
    <!-- the following line is necessary for some xform parsers -->
    <root><item><name/><label/></item></root>
</instance>

It's just a temporary dirty trick to fool ODK Validate, but it should work for all Pyxform-produced forms. Luckily the javarosa parser doesn't check the content of the choice_filter.

MartijnR added a commit to enketo/enketo-core that referenced this issue Apr 9, 2015

@MartijnR

This comment has been minimized.

Copy link
Contributor

MartijnR commented Jun 15, 2015

Bump @dorey, @ukanga, @tinok, @mberg. Who is going to add this to pyxform?

Currently this Enketo feature is only supported by ODK Aggregate.

@mberg

This comment has been minimized.

Copy link

mberg commented Jun 15, 2015

What's needed? Is there a ticket for this?

On Monday, June 15, 2015, Martijn van de Rijdt notifications@github.com
wrote:

Bump @dorey https://github.com/dorey, @ukanga
https://github.com/ukanga, @tinok https://github.com/tinok, @mberg
https://github.com/mberg. Who is going to add this to pyxform?

Currently this Enketo feature is only supported by ODK Aggregate.


Reply to this email directly or view it on GitHub
#30 (comment).

@dorey

This comment has been minimized.

Copy link
Contributor

dorey commented Jun 15, 2015

I wrote it out into a test here using the 'pyxform_test_case' syntax ( from pr #28 ).

@MartijnR, If that test describes all that's required, we can take a stab at it in the next couple weeks.

@MartijnR

This comment has been minimized.

Copy link
Contributor

MartijnR commented Jun 15, 2015

beautiful @dorey 👍 The only thing to change in the tests, is the ODK Validate trick, either by removing the self-closing / in the model_contains tests or by adding the <root><item><name/><label/></item></root> template. Former might be more attractive perhaps.

@mberg

This comment has been minimized.

Copy link

mberg commented Jul 2, 2015

Martijn,

Would this enable pull csv like functionality in enketo express?

Matt

On Mon, Jun 15, 2015 at 10:25 PM, Martijn van de Rijdt <
notifications@github.com> wrote:

beautiful @dorey https://github.com/dorey [image: 👍] The only thing
to change in the tests, is the ODK Validate trick, either by removing the
self-closing / in the model_contains tests or by adding the
template. Former might be more
attractive perhaps.


Reply to this email directly or view it on GitHub
#30 (comment).

@MartijnR

This comment has been minimized.

Copy link
Contributor

MartijnR commented Jul 2, 2015

[removed earlier reply]

pulldata() support has now been added to both Enketo (and pyxform) by adding the external <instance> with a src attribute.

@MartijnR

This comment has been minimized.

Copy link
Contributor

MartijnR commented Jan 4, 2016

UPDATED: ignore this comment

I wonder if, after the recent changes wrt to column 'value' we should output instead:

<value ref="value"/> <!-- ref="value" instead of "name" -->
<label ref="label"/>

This would change the ODK-validate trick into:

<instance id="cities" src="jr://file/cities.xml" >
    <!-- the following line is necessary for some xform parsers -->
    <root><item><value/><label/></item></root>
</instance>

and thus require a value and label column (csv) or node (XML) in the external document.

@MartijnR

This comment has been minimized.

Copy link
Contributor

MartijnR commented Aug 3, 2016

Actually "value" instead of "name" doesn't seem to work in PyxForm (for regular choices sheet). So let's stick with using "name". In any case, any OpenRosa XForm client supporting this feature could deal with both as it's just regular XForm syntax. So it could quite easily be changed later, without breaking existing forms.

The snippets in the OP are correct.

Sample XForm + XML media: external_xml.zip and live
Sample XForm + CSV media: external_csv.zip and live

@ukanga

This comment has been minimized.

Copy link
Contributor

ukanga commented Apr 4, 2017

@MartijnR I notice that the form.xml in external_xml.zip makes use of jr://file/cities.xml and what pyxform currently generates is jr://file-xml/cities.xml. Is this going to be an issue?

Also, what content type does enketo expect? The XML example https://enketo-stage.ona.io/::YYmJ seems Enketo is attempting a CSV processing instead of XML.

@MartijnR

This comment has been minimized.

Copy link
Contributor

MartijnR commented Apr 4, 2017

@ukanga, when looking at the code only, it seems like Enketo may (accidentally) be lenient towards this. However, jr:file-xml is definitely not correct. Sorry I missed this. For XML files it should be jr://file/cities.xml. (PS. this has now become part of the official ODK XForms spec)

Content-type should be appropriate for the source. At the moment, in Enketo if the (jquery get) auto-parsed result is not an XML Document it will be considered CSV. I think generally it should be text/xml or application/xml for XML and text/csv for CSV.

@MartijnR

This comment has been minimized.

Copy link
Contributor

MartijnR commented Apr 4, 2017

I've added an issue to EE as the explicitness of the jr:file or jr:file-csv connectors should probably be used to parse correctly regardless of the Content-Type it is served with. kobotoolbox/enketo-express#710

@ukanga

This comment has been minimized.

Copy link
Contributor

ukanga commented Apr 4, 2017

Thanks @MartijnR. I hope this is appropriate then #106.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment