In need of a Source object

[EssyGreen]

I realise the infinite flexibility of inheriting everything from a Resource and hence allowing it to be considered a source but this also makes for infinite complexity and infinite nonsense!

To allow, for example, a DatePart to be used as evidence is nonsensical. Theoretically we could cite millions of documents with a DatePart of "Day=1" but it means absolutely nothing without the wider context of the Date, which means nothing without the wider context of the Fact.

I think we need a definite Source object (probably equating, or similar, to the Record) and it is this (not the Resource) which should be referenced in Citations and used in Evidence.

[stoicflame]

There already is a Source object, although we already recognize it's inadequate.

But I think what you want is the requirement for all source references to resolve to an instance of that thing?

[jralls]

Just a different facet of #146

[EssyGreen]

There already is a Source object

Forgive me for being dim but the link just shows the "Description" with all the DC meta data .... I get that the DC meta data is effectively attributes/properties of the "Description" but where in the model is the "Description" object (which you say equates to a Source)?

I think what you want is the requirement for all source references to resolve to an instance of that thing?

Er, yes. How can they not? Surely all references to the same source should be links to the same Source object?

[stoicflame]

Just a different facet of #146

Yeah, that was my thoughts, too. I'm trying to figure out the difference.

but where in the model is the "Description" object

Umm... sorry... I don't understand the question. Where in the model? It's part of the model...

How can they not?

You could refer to an image as a source. Or a multimedia file. Or a web page. Or anything else that can be identified with a URI.

Surely all references to the same source should be links to the same Source object?

Sure. But not everything cited as a source needs to be an instance of that same type.

[jralls]

Or anything else that can be identified with a URI.

And that is the problem. As the model is presently expressed every reference reduces to a URI. I could enter a standard place, or some persona, or some random Slashdot article as a source. OK, a Slashdot article might be a valid source. Unlikely but possible. The real problem would be an internal object, an easy error to create, and which might cause actual trouble (think a is a source of b is a source of c is a source of a = crash).

[EssyGreen]

@stoicflame

There already is a Source object, [...]

I asked where and you gave me a link to a "Description" definition (http://www.gedcomx.org/model/dcterms_Description.html). Maybe it's just the .Net version is broken or something but there RDFDescription is never referenced anywhere so I can only assume its just the meta data for the whole GEDCOMX file ... ergo the file itself is the one and only source and all sources must be in separate GEDCOMX files referenced by their URIs.

If this is the case then as @jralls says any uri can be used as a Source and if that is so then there is no guarantee that there is any useful data whatsoever in the related uri since there is no guarantee it actually is a GEDCOMX file (and not just a jpg, a PC application, a link to a virus, etc).

[stoicflame]

And that is the problem. As the model is presently expressed every reference reduces to a URI.

Okay. How else would you suggest the serialization format (de)reference other objects? simple string?

The real problem would be an internal object, an easy error to create, and which might cause actual trouble (think a is a source of b is a source of c is a source of a = crash).

So that's an error in the data. Agreed.

How is that any different from any other data error cases that will need to be intelligently handled by the application? I can write an "I'm my own grandpa" loop, too.

[stoicflame]

ergo the file itself is the one and only source and all sources must be in separate GEDCOMX files referenced by their URIs.

No... that's not the intent of that at all. The intent of that object was to be akin to a Source object like you're asking for here in this thread. Obviously, that needs to be clarified, so I'll use this issue to track that work.

[jralls]

And that is the problem. As the model is presently expressed every reference reduces to a URI.

Okay. How else would you suggest the serialization format (de)reference other objects? simple string?

See #146. But:

The serialization isn't the model. The model describes the internal data structures that are serialized and that result when the stream is deserialized. So in the model references to other objects in the serial stream should be references to the class of those other objects, not the URI that is the proxy for the reference in the stream. Deserialization will have to be two passes: The first to construct all of the objects in the stream and the second to resolve the URIs into references and validate them.

[EssyGreen]

The serialization isn't the model. The model describes the internal data structures that are serialized and that result when the stream is deserialized. So in the model references to other objects in the serial stream should be references to the class of those other objects, not the URI that is the proxy for the reference in the stream.

+++++++++1

[jralls]

Deserialization will have to be two passes: The first to construct all of the objects in the stream and the second to resolve the URIs into references and validate them.

If the file format includes an index of the objects with their types as @nealmcb is asking for in #140 the first pass wouldn't be necessary.

(Yeah, I'm talking to myself. ;-) )

[EssyGreen]

LOL!

[stoicflame]

(I know I'm attempting to revive a stale thread here.)

Given the new set of specifications and recipes that attempt to clarify how to model the "source object" you seek, what still needs to be addressed here?

[EssyGreen]

Just catching up ... may take a while .... bear with me!

[jralls]

Given the new set of specifications and recipes that attempt to clarify how to model the "source object" you seek,
what still needs to be addressed here?

See #164 & #165

[EssyGreen]

Given the new set of specifications and recipes that attempt to clarify how to model the "source object" you seek, what still needs to be addressed here?

I'm still wading through the amended spec having been awol for some months but as far as I can see there is still no source object - just a "SourceReference" with an id, type, description and attribution. If the same source is reference multiple times then I presume these will also be replicated throughout the file creating a fragmented but inadequate source "record".

[jralls]

Over in #134 Sarah and I got going on Source analysis and its importance to good genealogical work.

In #156, Ryan expressed the mission of GedcomX:

The purpose of GEDCOM X has been stated as:

To define an open data model and an open serialization format for exchanging the components of the genealogical proof standard.

When we talk about "the components of the genealogical proof standard," we mean these:

• Search Reliable Sources
• Cite Each Source
• Analyze Sources, Information, and Evidence
• Resolve Conflicts
• Make a Soundly-Reasoned Conclusion

Which also recognizes the importance of source analysis. How then should GedcomX record the source analysis? The logical answer to me is to have a proper Source object with a citation property (what's currently called a "Description") and an analysis property (which can be just a long string).

[EssyGreen]

Can you clarify what you mean by "source analysis"? Is this what I call an "interpretation" ie working out what is explicitly and implicitly detailed in a single source?

[jralls]

Source analysis has three phases:

• Analyze the source itself: The provenance, the informant, whether the source is contemporary to the events or characteristics it records, how legible it is, etc.and through that analysis make an evaluation of the reliability of each "evidence statement" contained in the souce.
• Extract the evidence which is directly stated in the document.
• Analyze what the source says in the context of your understanding of the requirements and customs which govern similar sources from that place and time to infer indirect evidence. If the source is part of a larger source (e.g., a single household in a census), consider it in the context of the surrounding entries for more potential inferences. Pay attention as well for information which one ordinarily finds in the type of source at hand but which are missing from this one.

It's important to not try to make connections to evidence from other sources while you're doing this analysis so that you don't read in something that isn't there -- or miss something that is -- because "pieces of the puzzle" seem to fit.

Yes, you could call it interpretation if you like, but "working out what is explicitly and implicitly detailed" leaves out the first part.

[EssyGreen]

Yes that's what I call interpretation :)
I prefer to model this in the same way as in the Conclusion model rather than a single text description ... the only difference between them is the number of sources being analysed.

[jralls]

OK. How would you structure the Source class and how would you tie the elements into the conclusional Persons, Relationshps, and Events?

[EssyGreen]

Briefly - simplistic syntax:

Source is top level entity with properties:

• Persons
• Relationships
• Events
• Evidence (=Sources)
• Proof (text)

Quick example:
Source 1: Marriage Cert for Fred Bloggs & Freda Jacobs, GRO ref 1234ABC etc etc

• Person 1: Fred Bloggs
• Person 2: Freda Smith
• Relationships, Marriage event etc
• Proof: Original copy held by Boris Biggles - as descendant of Fred Bloggs
• Notes: Fred & Freda sign using their "mark"

Source 2: Birth Cert for Fred Bloggs, GRO ref 1234XYZ etc etc

• Person 1: Fred Bill Blogs
• Person 2: Freda James
• Person 3: Frederick Blogs
• Relationships, Birth event etc
• Evidence: ANOther Source
• Proof: Derivative from ANOther Source

Source 3: Birth Cert for Fred Bloggs, GRO ref 1234ABC etc etc

• Person 1: Fred Bill Blogs
• Person 2: Freda Smith
• Person 3: Frederick Blogs
• Relationships, Birth event etc
• Proof: Certified copy with photocopy from original register

Source 4: My Family Tree, author: Sarah Green other attributes of source etc etc

• Person 1: Fred Bill Bloggs
• Person 2: Freda Smith
• Person 3: Frederick Bloggs
• Relationships, Birth event, Marriage event etc
• Evidence: Source 1 (+5), Source 2 (-4), Source 3 (+4)
• Proof: The marriage cert. is considered to be accurate since it is held in the family records. Two births were found for Fred - the first (1234XYZ) is thought to be inaccurate since the mother's maiden name is James whereas the second (1234ABC) matches that of the marriage certificate. The slight difference in the spelling of the names is considered insignificant since neither Fred nor Fred were literate (as seen from the "Mark" on their marriage certificate)

NB: Persons are contained within each source, not pointers to somewhere else

[EssyGreen]

Hmm actually that's not quite right ... The proof of Source 4 should actually be the proof for Fred (or if you like for Fred's birth event) not the proof of the whole tree! Apologies.

[jralls]

OK.
Are the Persons, Relationshps, and Events objects in their own right or are they just strings? Yes, I saw the note at the bottom about them being contained, but they could still be structures with elements like Name, Age, Sex, etc. for the Persons.
Is "Evidence" the source-citation data (what the present spec calls the Description)?
Should Sources 1 - 3 have an Event, since that's what they seem to describe?
Shouldn't the relationships be "Person1 married Person2" (Source 1) and "Person3 child of Person1, Person3 child of Person2" (Sources 2 and 3)?

Isn't "Source 4" really a set of conclusion-model objects (Two Events, the marriage and the birth, 3 Persons, Fred, Freda, and Frederick, and 3 Relationships), each of which has the appropriate SourceReferences pointing back to Sources 1 - 3, along with the appropriate proof statements? (Here I'm assuming that you're not using your "My Family Tree" database as a source for some other database.)

Where would you put "Facts" (for example, the ages of the bride and groom from the marriage certificate)?

[EssyGreen]

Are the Persons, Relationshps, and Events objects in their own right or are they just strings?
Objects - I just used strings to simplify syntax here

they could still be structures with elements like Name, Age, Sex, etc. for the Persons.
Yup - again simplified for brevity of example

Is "Evidence" the source-citation data (what the present spec calls the Description)?
Probably similar - tho' I don't really understand what the GEDCOM X Description is or what it's trying to do

Should Sources 1 - 3 have an Event, since that's what they seem to describe?
Yup they can have events but the evaluation in the example is being done against the whole set not just any one single event so as to retain the context (i.e. Fred's birth place etc is intrinsically linked to the Father's name and occupation within the source so I wouldn't want to be able to cite one and quietly drop the other).

Shouldn't the relationships be "Person1 married Person2" (Source 1) and "Person3 child of Person1, Person3 child of Person2" (Sources 2 and 3)?
I didn't actually detail the relationships for simplicity but in my world they would be Source 1 (man/wife with marriage event & roles etc), Source 2/3: ditto plus child/mother and child/father

Isn't "Source 4" really a set of conclusion-model objects

If you're happy for the definition of a Source to be "a set of conclusion-model objects" then yes

I'm assuming that you're not using your "My Family Tree" database as a source for some other database

Why would you assume that? That's sort of my point that it is a source (albeit one being changed as the research progresses)

Where would you put "Facts" (for example, the ages of the bride and groom from the marriage certificate)?

• either as role of event or characteristics of person whichever you prefer :)

[thomast73]

I do not think a particular kind of hierarchy can be implied. All that can be said is that the sources listed in on description are related to the source described in in that description.

That's rather less than useless. What does "related to" mean in this context? What is the use of "SourceDerivation" if sources doesn't point to what it's derived from?

I am not suggesting that sources will be used to do anything that is not describing the provenance of the source being described. The reason we include sources in SourceDescription is so that we can provide the best possible description of a source`s provenance.

I do feel that the relationship between a source that is a "component of" another source is different than the relationship between a source that is "derived from" another source. It seems that the use cases proffered for the "component of" relationship are really about describing multiple elements from a single source -- a mechanism for defining a citation in pieces so that one or more of those pieces can be reused to in citing additional elements in that source.

On one hand, it seems the source metadata model is about supporting the need to describe a source and its provenance. This description (of the source and its provenance) is generally treated as a single logical item ("the source"). Regardless of the number elements used to describe "the source" (and the type of relationships that relate those elements), the user is still just describing "the source" and associating "the source" with his "conclusion".

I think the model is sufficient for this purpose.

On the other hand, if we were to state that the sources metadata model MUST support the automation of rending of entire provenance chains that include the citation of sources that are split into pieces (via the "component of" mechanism), then the model may need some help.

For example, it may be that if we added a type to SourceReference that could be either "derived from" or "component of", we could automatically combine consecutive instances of SourceDescription linked together by "component of" relationships into a single citation for that source, while citations for sources "derived from" other sources could be linked together with the " citing " phrase. But is this the right mechanism for this? I am not sure it is. I'd like to get a better feel for with the requirements for this really are. I think the model is better prepared to be morphed to support such scenarios. But we are not prepared to automate all of this yet.

[EssyGreen]

I don't agree with your use case for "component of". My intent (I believe I was the one to request it originally - see #123) was to enable the ability to represent the hierarchical nature of archival source collections and to cite from various parts of said collections. For example, a census book cover might have information about the households in statistical form and details about the enumerator. A page within that book might have details about a particular household. I may be interested in both and the fact that one is within the scope of the other may be relevant (e.g. the fact that the enumerator may have been recording his only family details; the fact that there should only be one representation of one person within a census etc etc).

If you are not prepared to provide a means of using these references then I think it is better and less confusing to simply omit them. After all, things may move on by the time you get to them (witness the old FONE and ROMN tags). Don't build in a partial "something" just in case it "might come in useful one day" - it will be ignored, misinterpreted and misused and we'll all have a nightmare trying to unravel the mess.

[jralls]

I believe I was the one to request it originally

No, I did.

For example, a census book cover might have information about the households in statistical form and details about the enumerator. A page within that book might have details about a particular household.

That's essentially the same use-case at a smaller scale:

• Census book SDA
• Book cover, SDB, is containedIn SDA
• Family page, SDC, is containedIn SDA.

(And book SDA is one of ten contained in box SDX, part of record group SDY).

If you are not prepared to provide a means of using these references then I think it is better and less confusing to simply omit them.

+1

[jralls]

I am not suggesting that sources will be used to do anything that is not describing the provenance of the source being described. The reason we include sources in SourceDescription is so that we can provide the best possible description of a source`s provenance.

OK, then the spec should say so, not just that the sources are "related to" the source being described.

On one hand, it seems the source metadata model is about supporting the need to describe a source and its provenance. This description (of the source and its provenance) is generally treated as a single logical item ("the source"). Regardless of the number elements used to describe "the source" (and the type of relationships that relate those elements), the user is still just describing "the source" and associating "the source" with his "conclusion".

OK again, but recognize that that will force a large amount of redundant data into the file and require extra parsing overhead on the part of using applications which do provide multi-level citations (and most provide for at least two levels).

[thomast73]

On one hand, it seems the source metadata model is about supporting the need to describe a source and its provenance. This description (of the source and its provenance) is generally treated as a single logical item ("the source"). Regardless of the number elements used to describe "the source" (and the type of relationships that relate those elements), the user is still just describing "the source" and associating "the source" with his "conclusion".

... but recognize that that will force a large amount of redundant data into the file and require extra parsing overhead ...

???

I am not suggesting we eliminate the ability to create a hierarchy of source descriptions to represent "the source". And I am not suggesting that parts of that hierarchy cannot be reused to describe another source. I am just saying that the whole chain of source descriptions (that starts with the reference in the conclusion) is logically one "source". If we used five source descriptions linked together to describe that source and its provenance, it is still logically one source.

[thomast73]

If you are not prepared to provide a means of using these references then I think it is better and less confusing to simply omit them.

What exactly is missing? How is it that there is no means of using these references?

[jralls]

If we used five source descriptions linked together to describe that source and its provenance, it is still logically one source.

Agreed.

I am not suggesting we eliminate the ability to create a hierarchy of source descriptions to represent "the source". And I am not suggesting that parts of that hierarchy cannot be reused to describe another source.

Then what do you mean by

I think the model is better prepared to be morphed to support such scenarios. But we are not prepared to automate all of this yet.

?

You earlier gave an example where you used the sources field to indicate a hierarchy, but at present there's nothing in the spec to indicate to an importing app that that's what they're there for. The SourceReferenceType did that, attaching meaning to each source.

[jralls]

What exactly is missing? How is it that there is no means of using these references?

A description of what each one of them is for. "related to" isn't sufficient, and "best possible description of a source's provenance" is pretty vague, too. SourceReferenceType described how each SourceReference in sources contributed to the "description of provenance". SourceDerivationType, being uncoupled from sources (never mind that their can be only one SourceDerivationType and many sources) doesn't:

The statement of derivation type is made independent of sources. I am not guaranteed to find a description of the source from which the described source was derived in the sources list, nor do I have a mechanism to identify it if it is in that list. The statement of derivation type merely represents that the source was analyzed and a judgment was made as to its relationship to its parent source (if applicable).

You can't seem to keep your story straight. ISTM you guys need to pull this and go think about it some more.

[stoicflame]

So what if SourceDescription contained a list of sources (as proposed) but also contained another property, componentOf that is of type SourceReference?

So then:

A. Family XX, page YY, accessed DD Month YYYY
B. Town, County, State, Roll
C. 1900 U.S. Census, digital image, FamilySearch
D. NARA Microfilm T623

Is modeled like:

Conclusion.sources = [ SrcRefA ]

SrcRefA.sourceDescription = SrcDescA

SrcDescA.citation= { …(Family XX, page YY, accessed DD Month YYYY)… }
SrcDescA.componentOf = SrcRefB

SrcRefB.sourceDescription = SrcDescB

SrcDescB.citation= { …(Town, County, State, Roll)… }
SrcDescB.componentOf = SrcRefC

SrcRefC.sourceDescription = SrcDescC

SrcDescC.citation = { …(1900 U.S. Census, digital image, FamilySearch)… }
SrcDescC.sources = [ SrcRefD ]
SrcDescC.derivationType = SourceDerivationType.PreservationCopy

SrcRefD.sourceDescription = SrcDescD

SrcDescD.citation = { …(NARA Microfilm T623)… }
SrcDescD.derivationType = SourceDerivationType.PreservationCopy

[stoicflame]

A description of what each one of them is for. "related to" isn't sufficient, and "best possible description of a source's provenance" is pretty vague, too. SourceReferenceType described how each SourceReference in sources contributed to the "description of provenance"

The problem is that we couldn't get very far down that road without confusing people. The question they kept asking went something like this:

If all source references to an extracted conclusion (or analysis, or transcription, or image, etc) are of type ExtractedConclusion, why don't you just put the type ExtractedConclusion on the description instead of repeating it all over the place?

Since we were unable to give a good answer, we decided to propose the consolidation of the type onto the SourceDescription.

So what would your answer to that question be?

[jralls]

So what if SourceDescription contained a list of sources (as proposed) but also contained another property, componentOf that is of type SourceReference?

I started to propose exactly that, but got distracted by Thad's "logically one source" tangent.

Yes, that will work.

[jralls]

So what would your answer to that question be?

I think I already did:

It's not a type of the Source, it's a usage type of the SourceReference.

So if SourceDescA describes a birth certificate, and EventA extracts the particulars of that birth event without any inferences, then the SourceReference in EventA pointing to SourceDescA has type "ExtractedConclusion". SourceDescB, which describes EventA, doesn't need a type, it has a pointer to the original. If AnalysisDocumentG uses SourceDescB, its SourceReference will be of type Analysis. There is no repetition.

[stoicflame]

And if AnalysisDocumentG uses all of SourceDescB, SourceDescC, SourceDescD, SourceDescE, and SourceDescF, under what circumstances will one of those SourceReferences not be of type Analysis? And if they're all of type Analysis, why do we even need a type on the source reference? You know it's of type Analysis because you're looking at an analysis document--the type is redundant and you're repeating yourself.

[jralls]

And if AnalysisDocumentG uses all of SourceDescB, SourceDescC, SourceDescD, SourceDescE, and SourceDescF, under what circumstances will one of those SourceReferences not be of type Analysis? And if they're all of type Analysis, why do we even need a type on the source reference? You know it's of type Analysis because you're looking at an analysis document--the type is redundant and you're repeating yourself.

Roger. So the only non-redundant usage is ExtractedConclusion vs. WorkingConclusion.

So rip it all out. SourceReference should be a SourceDescription Reference in the model, and the URI of a SourceDescription in the implementations. No need for typing, no need for a separate ID, no need for an Attribution.

SourceDescription doesn't need a SourceDerivationType, that's covered in the Citation, perhaps with additional detail provided in Notes.

SourceDescription does need a containedInSource SourceReference. I started to say that it needs a citesSource SourceReference too, but maybe that should be in the Citation as well. I could go either way on that.

Conclusion needs to have an "Extracted" flag.

[thomast73]

I started to say that it needs a citesSource SourceReference too, but ...

Isn't that what sources is? This case is already covered.

[jralls]

Isn't that what sources is? This case is already covered.

I don't know. Sometimes you say it is, sometimes not. If it is, SourceDescription should have only one -- only some Conclusion subclasses (AnalysisDocument, Event, Person, Relationship, maybe others, but only when ExtractedConclusion is False) need multiple sources.

But maybe, in the case of SourceDescription, it should be handled as part of Citation.

[thomast73]

But maybe, in the case of SourceDescription, it should be handled as part of Citation.

I certainly agree that it could be modeled that way. For now, lets get some feedback on the sources way of modeling it.

[thomast73]

So, to summarize, I will update the model as follows:

SourceReference

• sourceDescription : ResourceReference -- A pointer to the description of the source being referenced
• attribution : Attribution -- the attribution this reference

SourceDescription

• id : String -- a local-only, system identifier
• citation : SourceCitation -- the metadata used to described and cite this source
• about : URI -- if applicable, the URI to the actual source
• mediator : URI -- a reference to the entity that mediates access to the described source
• sources[0..*] : SourceReferences -- references to any sources from which this source is derived (i.e., sources that this source cites)
• componentOf : SourceReference -- a reference to the source that contains this source -- its parent context; for cases where this description is not complete without the description of its parent context
• displayName : String -- a display name for this source
• alternateNames[0..*] : TextValue -- a list of alternate display names for this source
• notes[0..*] : Note -- a list of notes about a source
• attribution : Attribution` -- the attribution for this source description

SourceDerivationType will be removed from the model.

NOTE: The "Extracted" flag will need to be discussed separately. We recognize the issue this flag attempts to address. We will open an issue after we merge this pull request.

NOTE: I am also making a change to the type of the alternateNames field; it is now of type TextValue; this is a slight simplification.

[jralls]

I think it would flow better if SourceReference were pulled up to 3.2 so that all of the CitationFoo paragraphs are together.

[EssyGreen]

sources[0..*] : SourceReferences -- references to any sources from which this source is derived (i.e., sources that this source cites)

I think this is confusing .... most people in genealogy would think of a citation as what we are now calling proof/evidence. You are using it to indicate a derivative which I think is a subtle but significant difference. For example, I "cite" a source in a narrative about someone; but a transcription of a baptism record "is derived from" the original.

Also I don't see the point of trying to list all the sources which cite/refer to this source since it will necessarily be impossible to do so in the real world. Instead each source should refer to the single source it is derived from (rather than trying to keep track of the infinite number of things which might reference it).

[EssyGreen]

I still don't see why we have to have an attribution in the SourceReference

[thomast73]

I have merged the pull request (#182) related to this issue. I am going to close this issue. I know that at least two questions are outstanding (about attribution - #192; and about ids - #198). If anything is unresolved and needs further discussion, would you please open a new issue(s) and summarize it there.

Thank you for all of the input given here! We appreciate the help in improving our model! :-)

[thomast73]

I forgot, there was one other issue that I promised to open as a result of conversation here -- relative to being able to distinguish "working conclusions" from "extracted conclusions". The issue can be found in #202.