export() returning improper data in 0.23.0

[zopyx]

{"_indexed":"2022-07-03T14:25:31.577647","all_paths":["/","/community","/community/projekt_db","/community/projekt_db/projekt_ecomediaeurope"],"created":"2007-09-04T12:45:14+00:00","description":"EcoMedia-Europa ist ein Sokrates-Comenius Netzwerk welches von der Europäischen Kommission kofinanziert wird. Das Hauptthema des Netzwerkes ist die strategische Entwicklung von IKT- und medienbasierter Bildung im Bereich von Sokrates-Comenius.","document_type_order":0,"effective":"1969-12-31T00:00:00+02:00","expires":"2499-12-31T00:00:00+02:00","headlines":"","id":"eteaching-8f5a8a4f71fa4583a194fea49923357b","id_original":"projekt_ecomediaeurope","language":"de","modified":"2015-04-08T09:42:52+00:00","path":"community/projekt_db/projekt_ecomediaeurope","portal_type":"Projekt","review_state":"published","subject":["comenius","sokrates","netzwerk"],"text":"http://www.ecomedia-europe.net/ hello world EcoMedia-Europa ist ein Sokrates-Comenius Netzwerk welches von der Europäischen Kommission kofinanziert wird. Das Hauptthema des Netzwerkes ist die strategische Entwicklung von IKT- und medienbasierter Bildung im Bereich von Sokrates-Comenius.","title":"EcoMedia Europe","uid":"8f5a8a4f71fa4583a194fea49923357b"}
{"_indexed":"2022-07-03T14:25:31.584181","all_paths":["/","/community","/![23710830-168f-43e4-af5f-279cb593c630](https://user-images.githubusercontent.com/594239/177045006-10958548-f951-460a-8b46-7c18c44cff7c.png)"],"created":"2007-09-10T09:21:48+00:00","description":"e-KoKon ist ein weitgreifendes dreigliedriges Maßnahmenpaket der Humboldt Universität Berlin, das die Aktivitäten der Hochschule zum Einsatz von E-Learning und Multimedia in der Lehre bündelt, indem dezentrale fachliche Kompetenz und zentrale Unterstützung zusammenführt werden. Auf diese Weise wird eine effektive Selbstorganisation der Fakultäten und Einrichtungen ermöglicht. Ein Netzwerk von E-Learning-Koordinator/innen in den Instituten stellt die Verbindung zwischen fachbezogenen Erfordernissen und zentral bereitgestellten Diensten her.

Using V 0.23.0, 0.14.0 of the Python.

I call export() on a collection. One particular record of the exported JSON is truncated. See above (last entry).

The content stored for this object can be seen in the screenshot - there is nothing obvious in the description field.

[jasonbosco]

Could you make sure the client-side timeout for the export API call is sufficiently large. It’s likely that the client is terminating the API call before the server response is fully sent to the client.

[zopyx]

I don't see any timeout parameter here:

https://typesense.org/docs/0.23.0/api/documents.html#export-documents

[jasonbosco]

This is the client-side timeout which is configured at the http library level. So if you’re using the PHP library for eg, which lets you bring your own http library, you want to look at the docs for the particular http library you’re using to configure the timeout.

[zopyx]

I am using Python. The export() call works without issues. If there would be a timeout error, a exception would be raised (which is not the case). The return result contains call 23.000 records from the collection..this particular one is broken.

[jasonbosco]

Oh I see that sounds odd then. So the record is not at the end of the response and somewhere in the middle?

[jasonbosco]

Could you also check if there’s a new line in the source document? Wondering if we need to escape new line chars inside docs when exporting

[zopyx]

wah...the issue is on the Python side...The export() result is correct. There is code on the Python level (in our module) that would use splitlines() for having one record per row. Unformatuntely the document contains U+2028 within the description which is also considered by splitlines() as a separator 😳
Sorry for the noise!