Skip to content

ConvoKit Version 3.0.0

Latest
Latest
Compare
Choose a tag to compare
@oscarso2000 oscarso2000 released this 26 Jul 02:41
· 32 commits to master since this release
v3.0.0
dbca34f
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

We're excited to announce the public release of Convokit 3.0!

The new version of ConvoKit now supports MongoDB as a backend choice for working with corpus data. This update provides several benefits, such as taking advantage of MongoDB's lazy loading to handle extremely large corpora, and ensuring resilience to unexpected crashes by continuously writing all changes to the database.

To learn more about using MongoDB as a backend choice, refer to our documentation at https://convokit.cornell.edu/documentation/storage_options.html.

Database Backend

Historically, ConvoKit allows you to work with conversational data directly in program memory through the Corpus class. Moreover, long term storage is provided by dumping the contents of a Corpus onto disk using the JSON format. This paradigm works well for distributing and storing static datasets, and for doing computations on conversational data that follow the pattern of doing computations on some or all of the data over a short time period and optionally storing these results on disk. For example, ConvoKit distributes datasets included with the library in JSON format, which you can load into program memory to explore and compute with.

In ConvoKit version 3.0.0, we introduce a new option for working with conversational data: the MongoDB backend. Consider a use case where you want to collect conversational data over a long time period and ensure you maintain a persistent representation of the dataset if your data collection program unexpectedly crashes. In the memory backend paradigm, this would require regularly dumping your corpus to JSON files, requiring repeated expensive write operations. On the other hand, with the new database backend, all your data is automatically saved for long term storage in the database as it is added to the corpus.

Documentation

Please refer to this database setup document to setup a mongoDB database and this storage document for a further explanation of how the database backend option works.

Tests

Updated tests to include db_mode testing.

Examples

Updated examples to include demonstration of db_mode usage.

Bug Fixes

  • Fixed issue where corpus.utterances throws an error in politenessAPI as it should call corpus.iter_utterances() instead. Corpus items should not access their private variables and should use the public "getters" for access.
  • Fixed bug in coordination.py for the usage of metadata mutability.
  • Fixed issue in Pairer with pair_mode set to maximize causing the pairing function to return an integer, which causes an error in pairing objects.

Breaking Changes

Modified ConvoKit.Metadata to disallow any mutability to metadata fields. Implemented by returning deepcopy of metadata field storage every time the field is accessed. It is intended to align the behaviors between memory and DB modes. #197

Change Log

Added:

  • Added DB backend mode to allow working with corpora using database as a supporting backend. #175 #184
  • Extended __init__ in model/corpus.py with parameters for DB functionality. #175
  • Updated model/backendMapper to separate memory and DB transactions. #175
  • Introduces a new layer of abstraction between Corpus components (Utterance, Speaker, Conversation, ConvoKitMeta) and concrete data mapping. Data mapping is now handled by a BackendMapper instance variable in the Corpus. #169

Changed:

  • Modified files in the ConvoKit model to support both memory mode and DB mode backends. #175
  • Removed deprecated arguments and functions from ConvoKit model. #176
  • Updated demo examples with older version of ConvoKit object references. #192

Fixed:

  • Fixed usage of the mutability of metadata within coordination.py. #197
  • Fixed issue in the Pairer module when pair_mode was set to maximize, causing the pairing function to return an integer and subsequently leading to an error. #197
  • Fixed issue that caused corpus.utterances to throw an error within politenessAPI. #170
  • Fixed FightingWords to allow overlapping classes. #189

Python Version Requirement Update:

  • With Python 3.7 reached EOL (end of life) on June 27, 2023, ConvoKit now requires Python 3.8 or above.
Assets 2