An open source document-oriented database that seeks to consolidate all primary sources related to Mormonism
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
1700
1800
1900
2000
venues
.gitignore
README.md

README.md

Mormon Primary Sources

With the technology of the twenty-first century, it's time to build an open source database that serves historians and other researchers in their study of primary source artifacts and documents. This project seeks to combine the advantages of document-based databasing, software versioning, and open source collaborative repositories to provide a scalable, searchable, and comprehensive record of the known body of primary sources. It takes Mormonism as its unifying topic, and will include a running collection of citations and meta-information for individual works and artifacts relating to this religious movement. The principal discipline around which this collection will be oriented is history, though other disciplines ought to benefit from the arrangement of these citations and whatever additional commentary that will be included with them. This collection will rely on guiding conventions to maintain the data and allow them to scale with new and changing methods of cataloging, systematizing, indexing, and searching. Contributors and users will need to be proficient in these conventions, which are outlined below.

Using This Database

In the near future, opensourcehistory.org will host this database. This site will provide a front-end interface for browsing and querying. While this website is in production, you can still make use of the database by first downloading it and then using a desktop search tool.

The easiest way to use this database is to download it using the GitHub interface. Under the "Downloads" tab, you can download a zip or tar.gz file containing all the files in this database. Any text editor will open these files for viewing or editing. Because this database is document-oriented, any number of desktop search tools can index and query this collection. On a Mac, simply open the parent Mormon-Primary Sources folder in a new Finder window and use the Spotlight search tool to limit the query to this folder. This technique is effective at providing comprehensive keyword search results without learning any query languages or interacting with a web page or other database browser. Other desktop search tools that work much the same way as Spotlight for the Windows platform include Google's Desktop Search, Windows 7's built-in desktop search tool, and others.

Contributing to This Database

You can contribute to this database by "forking" this repository. The simplest way to do this is on the Mac platform. On this page there is the "Clone in Mac" button. By clicking this, you will be taken to a page on GitHub where you can download their GitHub for Mac application. This application greatly simplifies the editing process. Because this database functions as a code repository, everything by every user is essentially saved to this repository without anything being lost. Every change that is "committed" is preserved. When you "clone" the repository on your computer, it will download the files to your desktop. If you make any changes to these files in the text editor of your choice, once you revisit GitHub for Mac, any changes you have made will be noted and you will have the ability to commit those differences to the repository. Your changes will not only be saved for your own use, but will be made available to the community of users that all contribute to this project. That's the magic of the open-source versioning model of collaboration. PC users can also contribute, but the process runs a little differently. Free resources are available at https://github.com/training/free and http://help.github.com/.

Collection Structure

This database is based on the document-oriented storage model (as opposed to the record-oriented or relational-oriented models). This means that the data is stored following a document structure: each item or record in the database is represented by a file housed within a folder, just like on a desktop platform. To facilitate browsing the collection, this project adopts a folder structure that behaves as a governing framework to the data store. Instead of a running list of records, a folder structure ordered by centuries, decades, and years breaks up historical time along a semantic trajectory. Arranging the folders this way also ensures that the date, whether a file is approximated to a year, decade, or century or not, behaves as a unique identifier to prevent data convergence or redundancy.

Some files will not be dated to a specific year; in these cases, the record would be housed in a corresponding decade. At worst, the record will be placed within an approimate century. Most often, database records will be placed in a corresponding year of its production and named in accordance with the file naming convention.

Folder and File Naming Conventions

To ensure that each document in the database represents an actual source, the following naming conventions are adhered to when committing records to the database. These naming conventions also allow for software indexers to be applied for abstracted searching and ordering of the database contents.

These records are primarily ordered by their date of production. The database, therefore, is only browsable by the chronological order of source materials. File names begin with the date then follow with the author, title, and in some cases the recipient. Each element of the filename is delimited by the underscore character. Multiple values within a naming element are delimited by the dash character. No spaces are allowed in the filename, and words are capitalized according to the Chicago Style parameters for headlines. For example:

1829-10-22_Smith-Joseph-Jr_Letter_Cowdery-Oliver.json

This file corresponds to the October 22, 1829 letter of Joseph Smith to Oliver Cowdery. The filename begins with the date, arranged first with the year, followed by the two-digit value for the month and the two-digit value for the day. Dashes separate the year from the month, and the month from the day. The underscore character delimits the date from the next element, which is the author. All personal names follow the lastname-firstname-additionalNames convention. In this case, the author is Joseph Smith. We enter his full name as Smith-Joseph-Jr and follow with another underscore to set off the next element. The title of the work is simply Letter following editorial convention. The final element in the filename is the recipient of the letter. A file extension .json sets this file apart as a JSON filetype. By following this naming convention, researchers can easily identify the document with the source it represents.

Folders follow three levels: century, decade, and year. Where records can only be associated with a decade or a century, files may be housed within the first two levels. Usually files will be housed at the year level. Occasionally it may be necessary to add another level of folders within the year level. These should correspond to the month at the fourth and the day at the fifth, and final, folder levels.

When naming fourth- or fifth-level folders, use the integer values for months and days (no leading zeros). Month or day levels should only be created when the number of files for a given year exceeds 500. Let's say that we have succeeded at committing all the documents for 1839. This number should be very high, considering the volume of documents within and without the Mormon community relating to the history of the Missouri period. We would create folders for 1 through 12 and within those folders place the documents that represent sources produced in each month.

Where documents are produced over the course of several days or months, the convention will be to assume the latest date in the production process for how to name and store the record.

Indexing Syntax

Additional data is provided inside each document file in the form of a JSON-formatted string. JSON is a robust yet simple format for encoding data. The basic JSON pattern is the key and value association. For example:

"name": "William McLellin"

The key in this example is "name" and the value is "William McLellin". Multiple keys and multiple values are stored using arrays and objects. Arrays are used to assign multiple values to a single key. Objects are used to define multiple keys. Arrays are encased in brackets. Objects are encased in curly braces, like so:

{
    "name": { "last": "McLellin", "first": "William", "middle": "Earl" },
    "family": [
        { "wife": { "married": "McLellin", "first": "Cinthia", "middle": "Ann" } },
        { "child": { "last": "McLellin", "first": "Charles", "middle": "William" } }
    ]
}

Here, a single object represents the whole record, which is why the document begins and ends with opening and closing curly braces, respectively. The first key in this object is "name", to which is associated another object which contains multiple keys: "last", "first", and "middle". Another key appears here after "name", so the "name" object ends with a comma to separate it from the next key. The second key is "family" to which is assigned an array. The array begins with an opening bracket, and each multiple value in this array will be separated by a comma. In this example, each value in the array is represented by an object that corresponds with each member of McLellin's family.

JSON allows for nested associations of keys and values, which is a major reason why it can scale so effectively with changing databases. Any key may have an array or object associated with it, and any array or object can contain any number of values.

For more examples of JSON in action, visit http://www.json.org/example.html.

Indexing Schema

The contents of each record file contain cataloging and researching information relating to the source. The following list of fields are used across the database. Additional keys may be used and not all keys will be present in each document. This facilitates scaling the data as the repository grows and the variability between sources increases. The document-oriented structure of this system encourages providing as much data as desired, but each data value must have a key. The Indexing Syntax section describes how the information is to be encoded to allow for easy parsing by search indexers.

/* Primary Information */

"name": [object]
    "last": [string]
    "first": [string]
    "middle": [string]
    "married": [string]
    "suffix": [string]
    "additional": [array]
    "patronymic": [string]
    "matronymic": [string]
    "maternalSurname": [string]
    "paternalSurname": [string]

"aliases": [array] [string or name object]

"date": [object]
    "year": [integer]
    "month": [integer]
    "day": [integer]

"dates": [array]
    "start": [date object]
    "end": [date object]

"author": [name object]

"authors": [array of name objects]

"editor": [name object]

"editors": [array of name objects]

"title": [string]

"subtitle": [string]

"series": [string]

"illustrator": [name object]

"illustrators": [array of name objects]

"translator": [name object]

"translators": [array of name objects]

"affiliation": [string or institution object]

"number": [integer]

"edition": [integer]

"city": [string]

"state": [string]

"country": [string]

"volume": [integer]

"issue": [integer]

"pages": [integer]

"firstEdition": [date object]

"volumes": [integer]

"venues": [array of venue objects]

/* Catalog Information */

"synopsis": [string]

"description": [string]

"notes": [string]

"submitted": [date object]

"reviewed": [date object]

"keywords": [array]

"label": [string]

"url": [string]

"doi": [string]

"isbn": [string]

"language": [string]

"status": [string]

"kind": [string]

"copyright": [object]
    "date": [date object]
    "holder": [string]

"loc": [string]

"access": [string]

"id": [string]

"license": [string]

"box": [integer | string]

"file": [integer | string]

"folio": [integer | string]

"path": [string]

Syntax Example

{
    "author": { "last": "Smith", "first": "Joseph", "suffix": "Jr." },
    "title": "Letter",
    "date": { "year": 1829, "month": 10, "day": 22 },
    "recipient": { "last": "Cowdery", "first": "Oliver" },
    "venues": [
        {
            "institution": "Church History Library",
            "path": "/venues/Church-History-Library.json",
            "collection": [
                "Joseph Smith Letterbook",
                "Joseph Smith Papers"
            ],
            "box": 1,
            "folio": 9
        },
        {
            "editor": { "last": "Vogel", "first": "Dan" },
            "author": { "last": "Vogel", "first": "Dan" },
            "title": "Early Mormon Documents",
            "volume": 1,
            "volumes": 5,
            "city": "Salt Lake City",
            "publisher": "Signature Books",
            "date": { "year": 1996 },
            "pages": [
                { "start": 7, "end": 8 }
            ]
        },
        {
            "title": "Joseph Smith Papers",
            "url": "http://josephsmithpapers.org/paperSummary/letter-to-oliver-cowdery-22-october-1829"
        }
    ]
}