Changelog

ConceptNet 5.5.0 (November 2016)

ConceptNet 5.5 contains many changes to make ConceptNet easier to work with and more powerful. The API has changed, but the data model is the same as ConceptNet 5.4.

Significant changes:

• A new Web API, with changes to make its results compatible with JSON-LD
• You can query by 'node' and 'other' in the API, matching edges connected to a node regardless of whether that node is considered the 'start' or 'end'
• New code for working with state-of-the-art word embeddings (conceptnet5.vectors)
• A new browsable Web interface, hosted at http://conceptnet.io
• Data is stored in PostgreSQL, providing reliable, fast queries
• Words can be connected to other Linked Data resources using the ExternalURL relation
• Words in all languages can be connected to their root form with the FormOf relation
• Because of that, English words no longer have to be reduced to a root form to be looked up. You can look up /c/en/words and get results.
• Converting text to its URI form is easy now: you lowercase the text and replace spaces with underscores. Languages such as Hindi are no longer mangled in URIs.

Data changes:

• Rewrote the Wiktionary parser to be more informative and accurate
• Added data from Open Multilingual WordNet
• Revised data from OpenCyc
• Rewrote the DBPedia reader to focus on terms that are also represented in other sources
• Removed GlobalMind as a data source. It helped us get started in various languages, but we aren't confident enough about its quality.

Process changes:

• Use Snakemake instead of Ninja to manage the build process
• Use Docker Compose to make the process of installing and building ConceptNet straightforward and reproducible

ConceptNet 5.4.2 (November 2015)

Bug fixes:

• The API server was resolving some URLs incorrectly and non-deterministically. Fixed the URL rules so that they don't overlap anymore.
• A wildcard match for * in the MANIFEST.in was matching directories, making setup.py install and pip install conceptnet crash.

ConceptNet 5.4.1 (September 2015)

This version fixes some bugs that existed in the build process of 5.4.0:

• Some of the raw data files that are needed to build ConceptNet were not being included in the distributed package of raw data.
• The data was unpacked into the wrong directory.
• The language codes 'zh-tw' and 'zh-cn' were not being recognized as being equivalent to 'zh'.

ConceptNet 5.4.0 (August 2015)

ConceptNet 5.4 includes small updates to the source data, a significant simplification to how texts are represented as URIs, and a new build process.

As of August 10, its code is available on the version5.4 branch, and its raw data is available at http://conceptnet5.media.mit.edu/downloads/v5.4/ . Other downloads and the Web API are still being prepared.

• We've updated the data from nadya.jp, a game that collects relational knowledge in Japanese.
• Updated DBPedia to its 2014 release.
• Dropped some not-very-useful relations that snuck in from old experiments with ConceptNet, such as /r/InheritsFrom.
• We've simplified the way that natural language texts are represented as ConceptNet URIs. Instead of an English-specific, machine-learned tokenizer from NLTK, we use a simpler regex based on Unicode character classes to split the text into words. The most noticeable change is that hyphens are now token boundaries just like spaces: /c/en/mother-in-law is now /c/en/mother_in_law, and the prefix /c/en/proto- is now /c/en/proto.
• Assertions now store the original texts of the terms that they relate in the surfaceStart and surfaceEnd fields. The assertion whose surfaceText is [[Fire]] is [[hot]] has properties including {"surfaceStart": "Fire", "surfaceEnd": "hot"}.
• The Makefile for ConceptNet was becoming unwieldy, so we've replaced it with a Ninja file. Ninja is a build system that's similar in spirit to Make, but deals better with parallel builds and with build steps that produce many outputs.
• Uses the langcodes module to parse language names and codes more robustly, especially those from Wiktionary.

ConceptNet 5.3.0 (November 6, 2014)

ConceptNet 5.3 introduces changes in many areas:

New API and search features

• The search index is now implemented in pure Python, using SQLite. Solr is no longer a dependency.
• The API now uses this new search index. One effect of this is that it matches only complete path components, not any prefix of a URI. Searching for "/c/en/cat" will get "/c/en/cat" and "/c/en/cat/n/animal", but not "/c/en/catamaran".
• Exact matches are also possible. Searching for "/c/en/cat/." -- with a dot as the final component -- will find only "/c/en/cat".
• Because the search index no longer uses Solr, the "score" attribute no longer appears on edges. This attribute was an artifact of Solr that represented the product of the edge's "weight" with Solr's built-in search weight. If you were using "score", you should use "weight" instead.

New and updated data

• ConceptNet now imports data from Umbel, a Semantic Web-style view of OpenCyc.
• Indonesian (id) and Malay (ms) concepts have been unified into the Malay macrolanguage (also designated ms), similarly to the way we already unify Chinese, because of their highly overlapping vocabularies. In a later version, we may be able to make distinctions between languages within a macrolanguage when necessary.
• We've implemented a better Wiktionary reader using Grako, a framework for writing recursive parsers in Python. This parser is able to understand the structure of a Wiktionary entry, giving more results and fewer errors than what we did before.
• Wiktionary parsing now covers entries written in German as well as English. (As before, the entries are about words in hundreds of languages.)

Edges in msgpack format

The intermediate format for lists of ConceptNet edges is now msgpack instead of JSON. This format is compatible with JSON but saves disk space and parsing time.

Updated assoc-space building

The "assoc-space", a dimensionality-reduced vector space of which words are like other words, uses an updated version of the assoc_space package. It can now be built in shards that are combined to form the complete space, instead of having to be built all at once, making it possible to run using a reasonable amount of RAM.

Getting a subset of ConceptNet under the CC-By license

Some of ConceptNet's data is available under the Creative Commons Attribution (CC-By) license, even though the dataset as a whole requires the Creative Commons Attribution-ShareAlike license (CC-By-SA). This information is marked on each edge, but in ConceptNet 5.2, there was no easy way to get the CC-By subset.

By now, there are enough CC-By-SA data sources that it doesn't make sense to attempt a complete build of ConceptNet without them. However, ConceptNet 5.3's downloads include a file containing only the CC-By edges, as individual edges that aren't grouped into assertions.

Deprecation of Python 2

ConceptNet 5.3's support code still runs on Python 2, but we would like to drop support for Python 2 in an upcoming version. As has been the case since version 5.2, the data cannot be built correctly on Python 2.

ConceptNet 5.2.4 (2014 August 29)

The data files described by MANIFEST.in are now also installed as package_data in setup.py, making them available when installed as a package. (This accomplishes what the last bullet point in 5.2.3 was supposed to be about.)

ConceptNet 5.2.3 (2014 June 27)

• Fix a typo in the Makefile that prevented it from downloading the initial raw data.
• Enforce the rate limit in the API.
• Merge in NLP code from metanl, instead of having it as an external dependency. The dependency is now on the simpler package ftfy.
• Add a MANIFEST.in so that the necessary data can still be found after a pip install or setup.py install.

ConceptNet 5.2.2 (2014 April 16)

• Fix the accidental omission of nadya.jp data.

ConceptNet 5.2.1 (2014 April 8)

5.2.1 is a significant revision to the code that builds ConceptNet, but it retains mostly the same representation and almost all of the same knowledge as 5.2.0. The cases where they differ are largely due to bugs that were discovered in the refactor.

• Reorganized much of the code for working with nodes and edges in ConceptNet.
• The code is now designed for Python 3 as its primary environment. A small amount of compatibility code makes sure that it will still run on Python 2.7 as well, but it will not necessarily get the same results from all Unicode operations.
• Removed a fair amount of dead code.
• Added test cases that cover most of the code; removed tests for 5.0 that clearly wouldn't work anymore.
• Combined assertions (such as what the 5.2 API returns) keep track of their full list of sources and their first-seen dataset, so they can be searched like edges in 5.1.

A change will be noticeable in the Web API, because for a while it was serving the union of ConceptNet 5.1 and 5.2 data structures, with both separate edges and combined assertions. Now it is only serving the combined assertions. The results should be similar, but with less duplication.

ConceptNet 5.2 (2013 September 17)

• The set of knowledge sources has changed. JMdict is in. ReVerb is out, because we couldn't filter it well enough.
• Some bugs in building from existing sources were fixed.
• ConceptNet can now be built from its raw data using a Makefile. (See Build process)
• The code comes with everything you need to build and query "assoc spaces" -- vector spaces representing semantic connections between concepts -- thanks to the open-source release of assoc_space by Luminoso.
• The API now returns one result per assertion, even if that assertion comes from multiple sources.
• Because of that, the representation of knowledge sources has changed. The sources used to be lists of reasons that an assertion got added, and each one implicitly represented a conjunction. The "sources" field in the API now always contains one element for each assertion, and that element contains the full AND-OR tree of sources.

ConceptNet 5.1 (2012 April 30)

Version 5.1 has a new, simpler representation of nodes and edges than ConceptNet 4.0 or 5.0, making it suitable to represent ConceptNet 5 with downloadable flat files and efficient search indexes.

• Made base URIs shorter. For example, /concept/en/dog becomes /c/en/dog.
• Changed the representation of assertions. Assertions are a bundle of edges (hyperedges, really) that connect two arguments and a relation. These edges are labeled with all the appropriate metadata.
• Created JSON and CSV flat-files.
• Created a Solr index and an accompanying API. The MongoDB is deprecated.

ConceptNet 5.1.1 was an incremental update that maintains full API compatibility with 5.1.

ConceptNet 5.0 (2011 October 28)

• First API for ConceptNet 5.
• All assertions were reified as nodes, with edges for arguments. This turned out to be an ineffective representation.