Add specifying @language to expanded form #159

Closed
msporny opened this Issue Sep 5, 2012 · 18 comments

Comments

Projects
None yet
5 participants
@msporny
Owner

msporny commented Sep 5, 2012

In order to fully support language-map round-tripping, we need to be able to express that a particular piece of markup in expanded form was specified using a language map. So, in order to round-trip this language map:

{
  tags: {
    en: {
      '@id': 'http://example.com/tags/foo', 'label': ' Foo'}
    de: {
      '@id': 'http://example.com/tags/baz', 'label': ' Baz'}
    }
}

We need to support this in expanded form:

{
  'http://example.org/vocab#tags': [{
    '@language': 'en', '@id': 'http://example.com/tags/foo', 'label': ' Foo' 
  }, {
    '@language': 'de', '@id': 'http://example.com/tags/baz', 'label': ' Baz'
  }]
}

To get back to this in compacted form:

{
  tags: {
    en: {
      '@id': 'http://example.com/tags/foo', 'label': ' Foo'}
    de: {
      '@id': 'http://example.com/tags/baz', 'label': ' Baz'}
    }
}
@lanthaler

This comment has been minimized.

Show comment Hide comment
@lanthaler

lanthaler Sep 6, 2012

Member

Sorry Manu, but I fear your proposal wouldn't work in this form. We will run into problems with @value objects that are not language-tagged strings. For example an integer would need to get an @language as well, so do typed values (meaning that you can't distinguish any more between a typed value and a language-tagged literal without relying on precedence rules).

The other problem I see is that if we allow completely arbitrary values in language maps, one could even include a language-tagged string which is tagged with another language than the one defined by the language-map and consequently round-tripping would break.

The only possible solution I see right now is to add a "@context": { "@language": __ } to all expanded values which would just express the default language... but that would be very weird because the fundamental functionality of expansion is to remove contexts altogether.

Nitpick: In the example above, after expansion, the value of "label" should be language-tagged strings.

Member

lanthaler commented Sep 6, 2012

Sorry Manu, but I fear your proposal wouldn't work in this form. We will run into problems with @value objects that are not language-tagged strings. For example an integer would need to get an @language as well, so do typed values (meaning that you can't distinguish any more between a typed value and a language-tagged literal without relying on precedence rules).

The other problem I see is that if we allow completely arbitrary values in language maps, one could even include a language-tagged string which is tagged with another language than the one defined by the language-map and consequently round-tripping would break.

The only possible solution I see right now is to add a "@context": { "@language": __ } to all expanded values which would just express the default language... but that would be very weird because the fundamental functionality of expansion is to remove contexts altogether.

Nitpick: In the example above, after expansion, the value of "label" should be language-tagged strings.

@gkellogg

This comment has been minimized.

Show comment Hide comment
@gkellogg

gkellogg Sep 10, 2012

Owner

I actually think it can probably work. It involves allowing @language to be used within node definitions. When expanding, string values would be expanded to include the included @language, such as:

{
  'http://example.org/vocab#tags': [{
    '@language': 'en', '@id': 'http://example.com/tags/foo', 'label': {'@value': ' Foo', '@language': 'en'} 
  }, {
    '@language': 'de', '@id': 'http://example.com/tags/baz', 'label': {'@value': ' Baz', '@language': 'de'}
  }]
}

As long as expanding leads to expanded values (which it does), this should allow compacting by adding rules for a local @language, and keeping that when compacting recursive objects.

Adding a '@context" can't be done when expanding, obviously. Otherwise, I think it can probably work, or I'm not completely understanding @lanthaler's concerns.

Owner

gkellogg commented Sep 10, 2012

I actually think it can probably work. It involves allowing @language to be used within node definitions. When expanding, string values would be expanded to include the included @language, such as:

{
  'http://example.org/vocab#tags': [{
    '@language': 'en', '@id': 'http://example.com/tags/foo', 'label': {'@value': ' Foo', '@language': 'en'} 
  }, {
    '@language': 'de', '@id': 'http://example.com/tags/baz', 'label': {'@value': ' Baz', '@language': 'de'}
  }]
}

As long as expanding leads to expanded values (which it does), this should allow compacting by adding rules for a local @language, and keeping that when compacting recursive objects.

Adding a '@context" can't be done when expanding, obviously. Otherwise, I think it can probably work, or I'm not completely understanding @lanthaler's concerns.

@lanthaler

This comment has been minimized.

Show comment Hide comment
@lanthaler

lanthaler Sep 11, 2012

Member

I was concerned about documents like this:

{
  "lang-map": {
    "en": [
      { "@id": "http://example.com/tags/foo", "label": "Foo.en" },
      1,
      "Foo.en",
      { "@value": "Foo" },
      { "@value": "Foo.type", "@type": "someType" }
    ],
    "de": [
      { "@id": "http://example.com/tags/bar", "label": "Bar.de" },
      2,
      "Bar.de",
      { "@value": "Bar" },
      { "@value": "Bar.type", "@type": "someType" }
    ]
  }
}

After expansion this would become:

  "lang-map": [
    {
      "@language": "en",
      "@id": "http://example.com/tags/foo",
      "label": [ { "@value": "Foo.en", "@language": "en" } ]
    },
    { "@value": 1, ***"@language": "en"*** }  <-- language-tagged number (xsd:integer)
    { "@value": "Foo.en", "@language": "en" }
    { "@value": "Foo" },   <-- can't add "@language": "en"! 
    { "@value": "Foo.type", "@type": "someType", ***"@language": "en"*** }  <-- precedence!?
    {
      "@language": "de",
      "@id": "http://example.com/tags/bar",
      "label": [ { "@value": "Bar.de", "@language": "de" } ]
    },
    { "@value": 2, ***"@language": "de" }  <-- language-tagged number (xsd:integer)
    { "@value: "Bar.de", "@language": "de" },
    { "@value": "Bar" },  <-- can't add "@language": "de"!
    { "@value": "Bar.type", "@type": "someType", ***"@language": "de"*** } <-- precedence!?
  ]
Member

lanthaler commented Sep 11, 2012

I was concerned about documents like this:

{
  "lang-map": {
    "en": [
      { "@id": "http://example.com/tags/foo", "label": "Foo.en" },
      1,
      "Foo.en",
      { "@value": "Foo" },
      { "@value": "Foo.type", "@type": "someType" }
    ],
    "de": [
      { "@id": "http://example.com/tags/bar", "label": "Bar.de" },
      2,
      "Bar.de",
      { "@value": "Bar" },
      { "@value": "Bar.type", "@type": "someType" }
    ]
  }
}

After expansion this would become:

  "lang-map": [
    {
      "@language": "en",
      "@id": "http://example.com/tags/foo",
      "label": [ { "@value": "Foo.en", "@language": "en" } ]
    },
    { "@value": 1, ***"@language": "en"*** }  <-- language-tagged number (xsd:integer)
    { "@value": "Foo.en", "@language": "en" }
    { "@value": "Foo" },   <-- can't add "@language": "en"! 
    { "@value": "Foo.type", "@type": "someType", ***"@language": "en"*** }  <-- precedence!?
    {
      "@language": "de",
      "@id": "http://example.com/tags/bar",
      "label": [ { "@value": "Bar.de", "@language": "de" } ]
    },
    { "@value": 2, ***"@language": "de" }  <-- language-tagged number (xsd:integer)
    { "@value: "Bar.de", "@language": "de" },
    { "@value": "Bar" },  <-- can't add "@language": "de"!
    { "@value": "Bar.type", "@type": "someType", ***"@language": "de"*** } <-- precedence!?
  ]

lanthaler added a commit that referenced this issue Sep 19, 2012

Add language-map expansion test
This test assumes that the language is injected as the default language into the active context. Depending on the outcome of issue #159 this test might need to be updated.

This addresses #133.
@msporny

This comment has been minimized.

Show comment Hide comment
@msporny

msporny Sep 25, 2012

Owner

Current options under consideration by the group:

  • Ask Drupal to change the data model (non-starter),
  • Adopt a 1-to-1 mapping between compact/expanded form for language maps, (adds complexity to syntax)
  • Adopt a complex algorithm to reconstruct language maps from expanded form, (adds complexity to API, and may be non-deterministic)
  • Model the data using BCP47 language code IRIs. (problematic from an RDF data model standpoint)
Owner

msporny commented Sep 25, 2012

Current options under consideration by the group:

  • Ask Drupal to change the data model (non-starter),
  • Adopt a 1-to-1 mapping between compact/expanded form for language maps, (adds complexity to syntax)
  • Adopt a complex algorithm to reconstruct language maps from expanded form, (adds complexity to API, and may be non-deterministic)
  • Model the data using BCP47 language code IRIs. (problematic from an RDF data model standpoint)
@lanthaler

This comment has been minimized.

Show comment Hide comment
@lanthaler

lanthaler Oct 9, 2012

Member

ACTION: Gregg to write up language-map spec text for the JSON-LD API.

Member

lanthaler commented Oct 9, 2012

ACTION: Gregg to write up language-map spec text for the JSON-LD API.

@ghost ghost assigned gkellogg Oct 9, 2012

@gkellogg

This comment has been minimized.

Show comment Hide comment
@gkellogg

gkellogg Oct 9, 2012

Owner

There are two proposals for supporting round-tripping of language maps in JSON-LD. Language map syntax is supported using @container: @language in a context document as specified in issue #133. This issue relates to defining expansion for language map in the case where the value is not a simple string.

In the first proposal, expanded values (included typed values), node definitions and node references are given an "@language" key/value pair:

{
  "@context": {
    "name": {"@id": "http://schema.org/name", "@container": "@language"}
  },
  "@id": "http://buckingham.uk/queenie",
  "name": {
    "en": [
      "The Queen",
      88,
      {"@id": "http://example.com/the_queen"}
    ],
    "de": [
      "Die Königin",
      89,
      {"@id": "http://example.de/de_königin"}
    ]
  }
}

This expands to a single property, but all values have @language added, even node definitions and typed values. This allows compaction to reverse the operation and properly allocate values to language tags.

[{
  "@id": "http://buckingham.uk/queenie",
  "http://schema.org/name": [
    {"@value": "The Queen", "@language": "en"},
    {"@value": 88, "@language": "en"},
    {"@id": "http://example.com/the_queen", "@language": "en"},
    {"@value": "Die Königin", "@language": "de"},
    {"@value": 89, "@language": "de"},
    {"@id": "http://example.de/de_königin", "@language": "de"}
  ]
}]

A complication is that typed values (which may also include @type) now may also contain @language. This creates some complexity when selecting the term to apply to a typed value (i.e., one may also exist which is coerced to the datatype of the value), but as this is a corner-case which is an anti-pattern, it seems acceptable.

The alternative algorithm would use @langmap instead of @language for both node definitions and typed values:

[{
  "@id": "http://buckingham.uk/queenie",
  "http://schema.org/name": [
    {"@value": "The Queen", "@language": "en"},
    {"@value": 88, "@langmap": "en"},
    {"@id": "http://example.com/the_queen", "@langmap": "en"},
    {"@value": "Die Königin", "@language": "de"},
    {"@value": 89, "@langmap": "de"},
    {"@id": "http://example.de/de_königin", "@langmap": "de"}
  ]
}]

This removes the ambiguity, but adds a separate keyword principally to support anti-patterns.

The proposal is to implement the first solution.

PROPOSAL: When expanding language maps, add @language to all expanded values, even for data-typed values.

PROPOSAL: When compacting, allocate values to language map terms (if they exist) when they contain an @language key even if a term with datatype coercion matches the datatype of a value.

Owner

gkellogg commented Oct 9, 2012

There are two proposals for supporting round-tripping of language maps in JSON-LD. Language map syntax is supported using @container: @language in a context document as specified in issue #133. This issue relates to defining expansion for language map in the case where the value is not a simple string.

In the first proposal, expanded values (included typed values), node definitions and node references are given an "@language" key/value pair:

{
  "@context": {
    "name": {"@id": "http://schema.org/name", "@container": "@language"}
  },
  "@id": "http://buckingham.uk/queenie",
  "name": {
    "en": [
      "The Queen",
      88,
      {"@id": "http://example.com/the_queen"}
    ],
    "de": [
      "Die Königin",
      89,
      {"@id": "http://example.de/de_königin"}
    ]
  }
}

This expands to a single property, but all values have @language added, even node definitions and typed values. This allows compaction to reverse the operation and properly allocate values to language tags.

[{
  "@id": "http://buckingham.uk/queenie",
  "http://schema.org/name": [
    {"@value": "The Queen", "@language": "en"},
    {"@value": 88, "@language": "en"},
    {"@id": "http://example.com/the_queen", "@language": "en"},
    {"@value": "Die Königin", "@language": "de"},
    {"@value": 89, "@language": "de"},
    {"@id": "http://example.de/de_königin", "@language": "de"}
  ]
}]

A complication is that typed values (which may also include @type) now may also contain @language. This creates some complexity when selecting the term to apply to a typed value (i.e., one may also exist which is coerced to the datatype of the value), but as this is a corner-case which is an anti-pattern, it seems acceptable.

The alternative algorithm would use @langmap instead of @language for both node definitions and typed values:

[{
  "@id": "http://buckingham.uk/queenie",
  "http://schema.org/name": [
    {"@value": "The Queen", "@language": "en"},
    {"@value": 88, "@langmap": "en"},
    {"@id": "http://example.com/the_queen", "@langmap": "en"},
    {"@value": "Die Königin", "@language": "de"},
    {"@value": 89, "@langmap": "de"},
    {"@id": "http://example.de/de_königin", "@langmap": "de"}
  ]
}]

This removes the ambiguity, but adds a separate keyword principally to support anti-patterns.

The proposal is to implement the first solution.

PROPOSAL: When expanding language maps, add @language to all expanded values, even for data-typed values.

PROPOSAL: When compacting, allocate values to language map terms (if they exist) when they contain an @language key even if a term with datatype coercion matches the datatype of a value.

@lanthaler

This comment has been minimized.

Show comment Hide comment
@lanthaler

lanthaler Oct 9, 2012

Member

I'm fine with the proposals except the expanded/typed values corner-case, so this:

"name": {
    "en": [
      "A",
      1,
      true,

      { "@value": "B },
      { "@value": 2 },
      { "@value": false },
      { "@value": "C", "@type": "someType" },
      { "@value": 3, "@type": "xsd:integer" },
      { "@value": false, "@type": xsd:boolean" }
  ]
}

Should expand to

"http://schema.org/name": [
      { "@value": "A", "@language": "en" },
      { "@value": 1, "@language": "en" },
      { "@value": true, "@language": "en" },

      { "@value": "B },
      { "@value": 2 },
      { "@value": false },
      { "@value": "C", "@type": "someType" },
      { "@value": 3, "@type": "xsd:integer" },
      { "@value": false, "@type": xsd:boolean" }
}

This is completely in-line with how we treat values in expanded object form with normal properties. Name below won't be expanded to a language-tagged string:

{
  "@context": {
    "name": {"@id": "http://schema.org/name", "@language": "de"}
  },
  "name": { "@value": "plain literal" }
}

To be consistent, we should then also language-tag booleans and numbers for simple language-tagged properties.

PROPOSAL: When expanding language maps, add @language to all expanded values, _except for values that are already in expanded object form_.

Member

lanthaler commented Oct 9, 2012

I'm fine with the proposals except the expanded/typed values corner-case, so this:

"name": {
    "en": [
      "A",
      1,
      true,

      { "@value": "B },
      { "@value": 2 },
      { "@value": false },
      { "@value": "C", "@type": "someType" },
      { "@value": 3, "@type": "xsd:integer" },
      { "@value": false, "@type": xsd:boolean" }
  ]
}

Should expand to

"http://schema.org/name": [
      { "@value": "A", "@language": "en" },
      { "@value": 1, "@language": "en" },
      { "@value": true, "@language": "en" },

      { "@value": "B },
      { "@value": 2 },
      { "@value": false },
      { "@value": "C", "@type": "someType" },
      { "@value": 3, "@type": "xsd:integer" },
      { "@value": false, "@type": xsd:boolean" }
}

This is completely in-line with how we treat values in expanded object form with normal properties. Name below won't be expanded to a language-tagged string:

{
  "@context": {
    "name": {"@id": "http://schema.org/name", "@language": "de"}
  },
  "name": { "@value": "plain literal" }
}

To be consistent, we should then also language-tag booleans and numbers for simple language-tagged properties.

PROPOSAL: When expanding language maps, add @language to all expanded values, _except for values that are already in expanded object form_.

@gkellogg

This comment has been minimized.

Show comment Hide comment
@gkellogg

gkellogg Oct 10, 2012

Owner

Not adding @language to already expanded values is consistent with how we process things now, but it does mean that we can't round-trip this use. We need to carefully consider if we should continue this, and risk creating a round-tripping issue.

Owner

gkellogg commented Oct 10, 2012

Not adding @language to already expanded values is consistent with how we process things now, but it does mean that we can't round-trip this use. We need to carefully consider if we should continue this, and risk creating a round-tripping issue.

@lanthaler

This comment has been minimized.

Show comment Hide comment
@lanthaler

lanthaler Oct 10, 2012

Member

The same is true for simple language-tagged properties, values in expanded object form don't round-trip, and that's a feature™ :-)

{
  "@context": { "name": { "@id": "http://schema.org/name", "@language": "@en" } },
  "name": [
      "Language-tagged string",
      { "@value": "Plain literal" }
    ]
}

Otherwise there wouldn't be a way to express a plain literal in a language-tagged property. You would have to define another property or use the full IRI instead.

@linclark is it a requirement for Drupal that also things like

{
  "@context": { "name": { "@id": "http://schema.org/name", "@container": "@language" } },
  "name": {
     "en": [
      { "@value": "Plain literal" },
      { "@value": "Typed value", "@type": "someType" }
    ]
  }
}

(this is compact form, not expanded form) round-trip cleanly?

Member

lanthaler commented Oct 10, 2012

The same is true for simple language-tagged properties, values in expanded object form don't round-trip, and that's a feature™ :-)

{
  "@context": { "name": { "@id": "http://schema.org/name", "@language": "@en" } },
  "name": [
      "Language-tagged string",
      { "@value": "Plain literal" }
    ]
}

Otherwise there wouldn't be a way to express a plain literal in a language-tagged property. You would have to define another property or use the full IRI instead.

@linclark is it a requirement for Drupal that also things like

{
  "@context": { "name": { "@id": "http://schema.org/name", "@container": "@language" } },
  "name": {
     "en": [
      { "@value": "Plain literal" },
      { "@value": "Typed value", "@type": "someType" }
    ]
  }
}

(this is compact form, not expanded form) round-trip cleanly?

@msporny

This comment has been minimized.

Show comment Hide comment
@msporny

msporny Nov 19, 2012

Owner

I'm trying to figure out where we are with this issue. We agreed that we want the simpler form of language maps in JSON-LD 1.0 - so, basically, we support only this form of language map in JSON-LD 1.0:

With this context:

{
"prop": {
"@id": "http://example.org/vocab#prop",
"@container": "@language"
}
}

and this compact form:

{
"prop": {
"en": "Foo",
"fr": "Fou",
"jp": "ふ"
}
}

would expand to:

{
"http://example.org/vocab#prop": [
{"@value": "Foo", "@language": "en"},
{"@value": "Fou", "@language": "fr"},
{"@value": "ふ", "@language": "jp"}
]
}

PROPOSAL 4: JSON-LD 1.0 will support simple language maps. When using a language map and expanding, if the term's language key's value is not a simple string, the rule for using the language map does not apply (all language-map values get dropped). When compacting, if all statements in the list are not simple @value/@language objects, then the term that defines the language map does not match (the statements are kept in expanded form).

Owner

msporny commented Nov 19, 2012

I'm trying to figure out where we are with this issue. We agreed that we want the simpler form of language maps in JSON-LD 1.0 - so, basically, we support only this form of language map in JSON-LD 1.0:

With this context:

{
"prop": {
"@id": "http://example.org/vocab#prop",
"@container": "@language"
}
}

and this compact form:

{
"prop": {
"en": "Foo",
"fr": "Fou",
"jp": "ふ"
}
}

would expand to:

{
"http://example.org/vocab#prop": [
{"@value": "Foo", "@language": "en"},
{"@value": "Fou", "@language": "fr"},
{"@value": "ふ", "@language": "jp"}
]
}

PROPOSAL 4: JSON-LD 1.0 will support simple language maps. When using a language map and expanding, if the term's language key's value is not a simple string, the rule for using the language map does not apply (all language-map values get dropped). When compacting, if all statements in the list are not simple @value/@language objects, then the term that defines the language map does not match (the statements are kept in expanded form).

@lanthaler

This comment has been minimized.

Show comment Hide comment
@lanthaler

lanthaler Nov 19, 2012

Member

Yes, this is basically what we came up with for #133:

RESOLVED: The values of the key-value pairs of a language map MUST be strings or arrays of strings. When expanded, the strings are tagged with the language specified by the key. When compacting, only language-tagged strings will match a term that has a "@container": "@language" mapping. Terms that have a "@container": "@language" mapping MUST NOT be type-coerced.

I would suggest we close this issue by simply referencing #133.

Member

lanthaler commented Nov 19, 2012

Yes, this is basically what we came up with for #133:

RESOLVED: The values of the key-value pairs of a language map MUST be strings or arrays of strings. When expanded, the strings are tagged with the language specified by the key. When compacting, only language-tagged strings will match a term that has a "@container": "@language" mapping. Terms that have a "@container": "@language" mapping MUST NOT be type-coerced.

I would suggest we close this issue by simply referencing #133.

@lanthaler

This comment has been minimized.

Show comment Hide comment
@lanthaler

lanthaler Nov 19, 2012

Member

PROPOSAL 5: Close by referencing #133: no further actions required.

Member

lanthaler commented Nov 19, 2012

PROPOSAL 5: Close by referencing #133: no further actions required.

@niklasl

This comment has been minimized.

Show comment Hide comment
@niklasl

niklasl Nov 19, 2012

Member

+1 to PROPOSAL 5.

Member

niklasl commented Nov 19, 2012

+1 to PROPOSAL 5.

@msporny

This comment has been minimized.

Show comment Hide comment
@msporny

msporny Nov 19, 2012

Owner

PROPOSAL 5: +1 (as long as we move the language over that I wrote above, it makes what we're doing clearer)

Owner

msporny commented Nov 19, 2012

PROPOSAL 5: +1 (as long as we move the language over that I wrote above, it makes what we're doing clearer)

@lanthaler

This comment has been minimized.

Show comment Hide comment
@lanthaler

lanthaler Nov 19, 2012

Member

PROPOSAL 5: +1

Member

lanthaler commented Nov 19, 2012

PROPOSAL 5: +1

@gkellogg

This comment has been minimized.

Show comment Hide comment
@gkellogg

gkellogg Nov 19, 2012

Owner

PROPOSAL 5: +1

Owner

gkellogg commented Nov 19, 2012

PROPOSAL 5: +1

@tidoust

This comment has been minimized.

Show comment Hide comment
@tidoust

tidoust Nov 20, 2012

Contributor

PROPOSAL 5: +1

Contributor

tidoust commented Nov 20, 2012

PROPOSAL 5: +1

@lanthaler

This comment has been minimized.

Show comment Hide comment
@lanthaler

lanthaler Nov 20, 2012

Member

RESOLVED: Close ISSUE-159 by referencing ISSUE-133: no further actions required.

Member

lanthaler commented Nov 20, 2012

RESOLVED: Close ISSUE-159 by referencing ISSUE-133: no further actions required.

@lanthaler lanthaler closed this Nov 20, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment