Check that recent ESSGlobal extensions are being read by sausage factory

[ColmMassey]

Are AM130, Am140, AM150, AM160 being read from Limesurvey forms?

It seems that for Liverpool Transport is not being read.

[ColmMassey]

What would it take for this to be automatic? Does something manual need to be done for each import script?

[ColmMassey]

Can you check this today? I'm hoping it will be an easy fix, at least for Liverpool. Need to get them running again. I have told them not to do any more durvey entries until this is resolved.

[wu-lee]

Conversed briefly with Dean earlier on Discord, only a few sentences, but I gather he's busy all today until 5ish.

I'd help but I haven't been involved with the work so far, so not clear on what to look at?

[wu-lee]

It seems that for Liverpool Transport is not being read.

Currently hunting for the problem (so I can track down the cause!)

I infer this means the "Transport" category is not being picked up somehow. And the AM-whatever IDs mentioned.

I'm looking in the Mersey data. Which initiatives specifically are being miscategorised/missed?

[ColmMassey]

Wirral Environmental Network listed Transport as a secondary activity. That is how the issue was noticed.

[wu-lee]

That initiative, ID 121, has these secondary activities if I regenerate the standard.csv data:

"Community & Collective Spaces;Education;Energy;Food;Nature, Conservation & Environment;Reduce, Reuse, Repair & Recycle"

So no transport. Looks like the converter scripts have these categories baked in, because the conversion depends on finding data in an arbitrary questionnaire format. Mersey uses the limesurveycore.rb module to convert, and that has yet to be updated.

[ColmMassey]

Have you looked in the raw csv from Limesurvey?

[wu-lee]

re. Automating this.

This is where the look ups are done for Mersey/Oxford/Newbridge:

https://github.com/SolidarityEconomyAssociation/open-data/blob/971a7ed88f8f9f35fe7fef5f9661dc0d59fa16a7/tools/se_open_data/lib/se_open_data/csv/converter/limesurveycore.rb#L136-L149

And this is the response for WEN.

https://solidarityeconomyassociation.limequery.com/admin/responses/sa/view/surveyid/201102/id/121/browselang/

Currently the secondary categories are checkboxes, with names and IDs which don't match the equivalent vocab ID in any way which can be inferred without some look-up index. If we want this to be automatic, we need ESS Global vocab terms in the results somewhere, or perhaps to autogenerate the look-up indexes somehow. Perhaps if we could name them like secondaryActivities_AM160 instead of secondaryActivities_SQ017 ...? But looking in the LimeSurvey in question, this doesn't seem to be possible.

[wu-lee]

Have you looked in the raw csv from Limesurvey?

Yes, unless I'm mistaken, that has a Y entry for the "Transport" option.

[ColmMassey]

Is Limesurveycore.rb the only place where you need to add the new terms?
Is the same code used for both Limesurveys?

If so, it's probably fine just to make the required changes here, unless you have a suggestion for automation?
I sense the time to do that might be when we are looking at how to do this one DigitalCommons/mykomap#62

[wu-lee]

Yes, this is the only place for data sourced from Lime Survey. I'll alter the hardwired index for now, I can't currently think of a nice way to automate it.

[ColmMassey]

Can you do this today? They want to populate it before the launch next week. When you do it canyou check it is rippling through to the dialog too?

[wu-lee]

Yes, hope to get it done today. Was trying to work out what the exact list of new terms was... turned out not to be so obvious which file was saying what, or why the makefile does what it does. However, having slept on it I think I should go and look at the lime survey questions. This may not be comprehensive, but it should be enough for Mersey.

Also, if you can point me at anything which can short-cut this search, also welcome.

[ColmMassey]

It is just AM130, Am140, AM150, AM160.
Agriculture, Industry, Utilities

& Transport

[ColmMassey]

They have direct analogies in Q7 & 8 in Liverpool limesurvey.

[ColmMassey]

FYI @wu-lee there may be someone doing a test survey entry on

https://solidarityeconomyassociation.limequery.com/201102?lang=en

today. Hope that doesn't mess with anything you are doing.

[wu-lee]

Should be fine.

[wu-lee]

Ok, so short version is, I think these values are propagating all the way through the sausage factory to the web page dialog now. I've checked Wirral Environmental Network on dev and prod, on both it has "Transport" as a secondary activity.

Some comments.

1. The map-ess repo has a bit of a strange branch structure (see here). Dean's most recent work is on the original repo in the branch extension-test. Whereas develop and master branches exist, but are way off somewhere else, and due to their age, I guess might be bit-rotten. We probably need to restore develop and master to appropriate places.

2. Clicking around the ESS Global website, I have the impression that some anchor links don't work properly. "Legal form" is there, but trying to find the list of terms, I don't seem to be able to get very far. The reason I was looking was I think that "organisational structure" is the replacement term for "legal form", and they both seem to be listed.

3. I found a list of "activities", in activities-modified.skos, with the new ones added here on extension-test, which correlate to the limesurvey questions. All good. But I notice that the list of "organisational structures" in the questionnaire is much shorter than the vocab defined in organisational-structure.skos, (also with new ones added here). Should these not be added to the questionnaires too?

4. Finally: having fixed this for Mersey, I find that this means that Oxford and Newbridge now have a different schema (I checked, and these new terms are not options in their secondary activities multiple-choice). So the list of required fields for Mersey includes fields Oxford and Newbridge don't. And this breaks their conversion, which is designed to catch mismatches like this and throw an error.

5. So I've added a commit b3a727a to work around that, but it breaks the intent to validate. I think this is what they call a "code smell": shouldn't all three have an identical survey schema?

[ColmMassey]

Ok, so short version is, I think these values are propagating all the way through the sausage factory to the web page dialog now. I've checked Wirral Environmental Network on dev and prod, on both it has "Transport" as a secondary activity.

Wirral Environmental Network isn't on the original Limesurvey?
https://solidarityeconomyassociation.limequery.com/admin/responses/sa/browse/surveyid/376295

Are prod and dev now both pointing to the newer one

https://solidarityeconomyassociation.limequery.com/201102?lang=en ?

[ColmMassey]

1. The [map-ess](https://github.com/SolidarityEconomyAssociation/map-sse) repo has a bit of a strange branch structure (see [here](https://github.com/SolidarityEconomyAssociation/map-sse/network)). Dean's most recent work is on the [original repo](https://github.com/essglobal-linked-open-data/map-sse) in the branch `extension-test`. Whereas `develop` and `master` branches exist, but are way off somewhere else, and due to their age, I guess might be bit-rotten. We probably need to restore `develop` and `master` to appropriate places.

It would be good to get clarity from @dtmakm27 as to the history of these branches restore to 'develop' and master.

[ColmMassey]

2\. Clicking around the ESS Global website, I have the impression that some anchor links don't work properly. "Legal form" is there, but trying to find the list of terms, I don't seem to be able to get very far. The reason I was looking was I think that "organisational structure" is the replacement term for "legal form", and they both seem to be listed.

I could definitley benefit from a tidy. There may already be a ticket to conduct a tidy. If you can't see one, please create this a new Issue.

[ColmMassey]

3\. But I notice that the list of "organisational structures" in the questionnaire is much shorter than the vocab defined in [organisational-structure.skos](https://github.com/essglobal-linked-open-data/map-sse/blob/db7aa357891a137b3e9e4b0576e478b97dbb8bd9/vocabs/standard/organisational-structure.skos), (also with new ones added [here](https://github.com/essglobal-linked-open-data/map-sse/commit/db7aa357891a137b3e9e4b0576e478b97dbb8bd9)). Should these not be added to the questionnaires too?

Not necessarily. It is fine to provide options in a survey which are only a subset of the possibilities. The full set may confuse some users. Next time we review the Oxford and Newbridge surveys we should consider extending.

[ColmMassey]

4\. Finally: having fixed this for Mersey, I find that this means that Oxford and Newbridge now have a different schema (I checked, and these new terms are not options in their secondary activities multiple-choice). So the list of required fields for Mersey includes fields Oxford and Newbridge don't. And this breaks their conversion, which is designed to catch mismatches like this and throw an error.

I guess this is because each answer option in those questions requires a new column, and then the schema defines these new columns as required?

[ColmMassey]

5\. shouldn't all three have an identical survey schema?

I think the issue is the schema is too rigidly structured becasue of how the Limesurvey export csv is structured. Worth a chat to talk through.

[wu-lee]

On 11/11/2020 11:31, Colm Massey wrote: |4\. Finally: having fixed this for Mersey, I find that this means that Oxford and Newbridge now have a different schema (I checked, and these new terms are not options in their secondary activities multiple-choice). So the list of required fields for Mersey includes fields Oxford and Newbridge don't. And this breaks their conversion, which is designed to catch mismatches like this and throw an error. | I guess this is because each answer option in those questions requires a new column, and then the schema defines these new columns as required?

Yes, each new column has to be added to the open-data incoming schema (schema.yml) for it to be noticed. Anything not in there is ignored by design. This is because all the New/Ox/Mer schemas are slightly different, but  include a common core (or *used to*). At one point any differences wouldn't be tolerated, but I loosened it to allow superfluous fields. N

[ColmMassey]

All looking good for Liverpool. Can you now change prod. to get its data from

https://solidarityeconomyassociation.limequery.com/201102?lang=en ?

[ColmMassey]

Wirral Environmental Network isn't on the original Limesurvey?
https://solidarityeconomyassociation.limequery.com/admin/responses/sa/browse/surveyid/376295

Are prod and dev now both pointing to the newer one

https://solidarityeconomyassociation.limequery.com/201102?lang=en ?

Ignore this comment. I don't know what happened but when I looked again, all seemed as it should have.

[wu-lee]

On 11/11/2020 13:43, Colm Massey wrote: Ignore this comment. I don't know what happened but when I looked again, all seemed as it should have.

I do see "Transport" included on WEN's dialog on https://prod.mersey-green.solidarityeconomy.coop, and this puzzled me, because when I checked, it is using the original #376295 survey. But then I see that the survey has had the Transport et. al options added to that question, which explains it.

[ColmMassey]

I have created new tickets for the issues raised in this ticket. Lets close this one once prod.mersey-green is referencing the new survey.

[wu-lee]

Ok, updated the staging.conf file to use survey 201102 and pushed. The sausage factory should handle the rest in a few minutes.