Refactoring and add language bindings

[nokome]

This PR addresses two items from the HackMD:

- Explore possibility of generating stricter TypeScript definition (remove `Node: any`)?
- Explore possibility of other language type e.g. Python, GraphQL

This was primarily started towards adding factory functions for Typescript (#84) but I have actually started with language bindings for other languages (as a means for user-developers to be able to generate document trees, that are type safe, from within their language of choice).

@alex-ketch Note that this makes some changes to generation .schema.json files that may affect documentation generation e.g. parent has been removed, use extends instead

[nokome]

Some options for discussion

/********************************************************
 * Not-extending interfaces (current) + factory functions
 *******************************************************/

// Interfaces generated by `json-schema-to-typescript` look something
// like this:

interface Thing1 {
  type: 'Thing' | 'Person'
  id?: string
}

interface Person1 {
  type: 'Person'
  id?: string
  givenNames: string[]
}

// A lot of our code looks like this with 'manual'
// creation of typed objects and assignment to
// optional properties.
const pat: Person1 = {
  type: 'Person',
  givenNames: ['Pat']
}
pat.id = 'pat'

// Factory functions would help to reduce this boilerplate code
// and could take the convention used for Python
// and R of putting properties as parameters (i.e. optional last)

function person1a(givenNames: string[], id?: string): Person1 {
  return {
    type: 'Person',
    id,
    givenNames
  }
}

const pam = person1a(['Pam'], 'pam')

// Having a long list of optional parameters is OK for Python and R because
// they have keyword arguments. For Typescript it would be better to have
// an `options` parameter.

function person1b(givenNames: string[], options: Partial<Person1> = {}): Person1 {
  return {
    ...options,
    type: 'Person',
    givenNames
  }
}

const pete = person1b(['Peter'], {id: 'pete'})

/********************************************************
 * Extending interfaces + factory functions
 * As above, but generating interfaces ourselves to take
 * advantage of `extends` keyword to reduce amount of generated code
 * (currently `types.d.ts` is over 3000 lines with lots of repetition)
 *******************************************************/

interface Thing2 {
  type: 'Thing' | 'Person'
  id?: string
}

interface Person2 extends Thing2 {
  type: 'Person'
  givenNames: string[]
}

/********************************************************
 * Classes
 *
 * Use classes to:
 *  - (a) have a constructor and thus avoid having both a
 *        interface (`Person`) and a function (`person`),
 *  - (b) make state mutable by using `readonly`
 *******************************************************/

class Thing3 {
  readonly type: 'Thing' | 'Person'
  readonly id?: string
}

class Person3 extends Thing3 {
  readonly type: 'Person'
  readonly givenNames: string[]

  constructor(givenNames: string[], options: Partial<Person3> = {}) {
    super()
    return {
      ...this,
      ...options,
      type: 'Person',
      givenNames
    }
  }
}

const paul = new Person3(['Paul'], {id: 'paul'})
// This gives a compiler error
paul.givenNames = ['Pat']

[nokome]

I've gone ahead and implemented the second option above.

@beneboy Could you npm run build:py and check the generated python/types.py for style and correctness. There is python.test.sh and tests/python.test.ts if you want to test any changes.

@alex-ketch Could you take over this PR. Some things that still need doing:

• npm run ts and check the generated dist/types.ts for style and correctness. The tests/typescript.test.ts build test should test the type correctness of that files.
• incorporate the generated dist/types.ts into the tsc compilation step

I suggest that we don't spend too much more time on this right now, but follow up tasks to do in the future include:

• providing usage guide docs for each language
• looking at the commonalities in python.ts. r.ts and typescript.ts and factoring those out to remove repetion of code - in particular schemaToType et al are ripe for that
• running R and Python tests as part of CI

[beneboy]

The building of Python (and other) type files was failing for me. Doing some debugging it looks like and issue when it's trying to render the JSONSchema schema which just seems wrong? I'm guessing if it works for you there's something gone funny with my schema not being generated properly before running the Python build.

Here's a snippet of the JSONSchema class that's getting generated. I have had schemaToType return Unhandled schema instead of throwing an error to get an idea where it's failing.

class JSONSchema:
    """
    undefined
    """

    def __init__(
        self,
        $comment: Optional[str] = None,
        $id: Optional[str] = None,
        $ref: Optional[str] = None,
        $schema: Optional[str] = None,
        additionalItems: Optional["#"] = None,
        additionalProperties: Optional["#"] = None,
        allOf: Optional["#/definitions/schemaArray"] = None,
        anyOf: Optional["#/definitions/schemaArray"] = None,
        const: Optional[Unhandled schema] = None,
        contains: Optional["#"] = None,
        contentEncoding: Optional[str] = None,
        contentMediaType: Optional[str] = None,
        default: Optional[Unhandled schema] = None,

Running make clean and then npm run build:py allows me to build without error and JSONSchema is no longer in the generated types.py.

There's a few things I've noticed that I'm going to clean up. The first is that the type properties are inherited so they don't need to be defined on a child class if the parent already has them. This seems to get done for some of the classes/properties but not all so I'll investigate this. There's a few small format things with extra new lines that I'll try to sort out too. Also if there is no documentation I'll make it not generate the docstring for the class.

[nokome]

Yeah, that's a hangover from when we had a schema/json-schema-version-7.schema.json files. That's now gone so the make clean was needed to remove built/json-schema-version-7.schema.json.

If you could fix up those things, that'd be great. Thanks.

[beneboy]

I fixed up what I could with formatting, flake8 now just complains about max line lengths which is a bit difficult to sort out with code formatting. It might be easier just to run the generated code through autopep8 which I've tried out and fixes the max line length issue. I don't know how high of a priority this is though given that the code is still valid even if it's not PEP8.

I will still need to investigate doubled up properties (parents and children). Case in point contentUrl on AudioObject and MediaObject.

[nokome]

I remembered why I didn't include a type attribute in Python classes: because it duplicates __class__.__name__ so felt a little redundant in that context. I thought we could add type from the class name when we do marshalling to dict/JSON like the approach at the end of https://medium.com/python-pandemonium/json-the-python-way-91aac95d4041

Yeah, let's not worry about PEP8 conformance right now.

The doubled up properties come from the need to override an optional property as being required in a derived class (for Python) / interface (in the case of Typescript). I hurriedly made the own properties be those that are not inherited or are not optional:

schema/src/bindings.ts

Line 109 in 1e7a6c0

own: props.filter(prop => !prop.inherited || !prop.optional),

We'd need to walk back through ancestors to do a better job of that. Probably best place to do that is at time of generating *.schema.json files since we are walking back through ancestors then:

schema/src/schema.ts

Lines 138 to 145 in 1e7a6c0

	// Add to all ancestors' descendants and type enum
	let ancestor: Schema | null = parent
	while (ancestor !== null) {
	ancestor.descendants =
	ancestor.descendants === undefined
	? [title]
	: [...ancestor.descendants, title]
	if (

[beneboy]

The Python convention is to not have the body of the if statement on the same line as the if. Also the else is missing a : at the end.

You could just convert this to a Python "ternary" though:

return fromDict(node) if isinstance(node, dict) else node

[nokome]

Hah, yeah I actually wrote the ternary first but then decided to be more conventional - and break it :)
Also, you missed the fact that I used json as a variable name which clashes with the module name 😊 Anyway, I've created #101 for you to clean up my mistakes :)

[nokome]

This branch has some things in it that are needed in the updates for Encoda (specifically, how schema.json files are distributed. So I'm going to resolve conflicts and merge now.

[stencila-ci]

🎉 This PR is included in version 0.17.0 🎉

The release is available on:

• GitHub release
• npm package (@latest dist-tag)

Your semantic-release bot 📦🚀