Full KML exports are broken

[paregorios]

Since at least 10 June 2024, all of the full Pleiades KML exports are only 172 bytes in size (i.e., the export process is erroring off before completion).

Individual KML exports seem to be OK in general. This is likely being caused by a single damaged Pleiades location, which we'll need to find.

Reported by Gabriel Mckee.

[paregorios]

But see also #510

[paregorios]

@jessesnyder unfortunately the fix to #510 does not seem to have resolved this problem. This morning's export appears to have errored off (see https://atlantides.org/downloads/pleiades/kml/).

[paregorios]

I suspect that a damaged published Location somewhere may be causing this problem, but we can't no for sure unless the export can be run manually and watched for tracebacks or other errors.

[jessesnyder]

Attempting to run this manually, I do get an immediate failure. I'm not sure this is the same error that's happening "in the wild", or if I'm missing some set up. It is surprising that anything impacting this would have changed in mid-June.

Command:

sudo -u plone_daemon bin/debug-instance run dump-scripts/kml.py

Excerpt of output:

Traceback (most recent call last):
  File "/srv/python27-apps/pleiades4/parts/debug-instance/bin/interpreter", line 338, in <module>
    exec(_val)
  File "<string>", line 1, in <module>
  File "dump-scripts/kml.py", line 14, in <module>
    macros={'kml_template': main_macros, 'kml_macros': kml_macros})
  File "/srv/python27-apps/pleiades4/eggs/zope.pagetemplate-3.6.3-py2.7.egg/zope/pagetemplate/pagetemplate.py", line 108, in __call__
    return self.pt_render(self.pt_getContext(args, kwargs))
  File "/srv/python27-apps/pleiades4/eggs/zope.pagetemplate-3.6.3-py2.7.egg/zope/pagetemplate/pagetemplate.py", line 132, in pt_render
    strictinsert=0, sourceAnnotations=sourceAnnotations
  File "/srv/python27-apps/pleiades4/eggs/zope.pagetemplate-3.6.3-py2.7.egg/zope/pagetemplate/pagetemplate.py", line 240, in __call__
    interpreter()
  File "/srv/python27-apps/pleiades4/eggs/zope.tal-3.5.2-py2.7.egg/zope/tal/talinterpreter.py", line 271, in __call__
    self.interpret(self.program)
  File "/srv/python27-apps/pleiades4/eggs/zope.tal-3.5.2-py2.7.egg/zope/tal/talinterpreter.py", line 343, in interpret
    handlers[opcode](self, args)
  File "/srv/python27-apps/pleiades4/eggs/zope.tal-3.5.2-py2.7.egg/zope/tal/talinterpreter.py", line 888, in do_useMacro
    self.interpret(macro)
...
...
  File "/srv/python27-apps/pleiades4/eggs/zope.tales-3.5.3-py2.7.egg/zope/tales/tales.py", line 696, in evaluate
    return expression(self)
  File "/srv/python27-apps/pleiades4/eggs/zope.tales-3.5.3-py2.7.egg/zope/tales/expressions.py", line 217, in __call__
    return self._eval(econtext)
  File "/srv/python27-apps/pleiades4/eggs/zope.tales-3.5.3-py2.7.egg/zope/tales/expressions.py", line 194, in _eval
    ob = self._subexprs[-1](econtext)
  File "/srv/python27-apps/pleiades4/eggs/zope.tales-3.5.3-py2.7.egg/zope/tales/expressions.py", line 118, in _eval
    ob = vars[base]
KeyError: 'folder'

[jessesnyder]

Looks like it could be this commit:
isawnyu/pleiades-kml@829a101#diff-8aa9ac99637da3507d3334056be3923b9e1495231a062d32b2857aa35356485aR17

I see that a reference to folder/name was added to a template used in the dump. This commit is 5 months old, but maybe some data change that exposed the problem later.

Off to appt... tbc.

[paregorios]

Right. This was an attempt to provide better description text in the KML to show up on the kite so we can direct people back to the relevant Pleiades place, via a link. Some questions:

• Any way to find out which place it's trying to process when it throws the error? Looking at it might help us figure out what is going on.
• Why is it fiddling with a "folder" instead of a "place"?

[jessesnyder]

@paregorios Can you elaborate on what you hoped folder/name would be? As far as I can tell, everything being output is a Place (not a Location), and they all live in /places/.

[jessesnyder]

Update is on staging, @paregorios. Remember to re-test individual KML exports to verify the compound naming still works in this case (I tested locally and it worked for me).

[paregorios]

@jessesnyder will do. Did you manually run the bulk export and, if so, where should I look for it on staging path-wise?

[jessesnyder]

I didn't but I can do that now... I'll report back w/ the path

[paregorios]

KML serializations of single place resources, reached via links like "https://example.org/places/12345/kml" are working fine on staging.

[jessesnyder]

/srv/python27-apps/pleiades4/var/514_test.kml

514_test.kml.zip

[paregorios]

Overall, this is now looking like what we've long expected. Here's an example pop-up in Google Earth:

@jessesnyder Can we use a similar quick test to eliminate the "In Context: Pleiades KML" line when we're doing the full dump, but keep the current behavior on the individual place serializations?

[jessesnyder]

Interestingly, that's added by the kml_topic_document.pt template, so we can't use the same test. I'd need to figure out how to determine the difference in context in that template. I'll see if I can figure this out quickly.

[jessesnyder]

@paregorios Deployed and ran the new code on production and the dump completed without errors. (I did not send the output to a file, so nothing to see, but it should run correctly from the cron job later).

[paregorios]

Fixed on production. Closing as resolved.