Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[epic] Resource Schema (data dictionary) editor w/ infer #13

Closed
17 of 18 tasks
rufuspollock opened this issue Aug 18, 2020 · 14 comments
Closed
17 of 18 tasks

[epic] Resource Schema (data dictionary) editor w/ infer #13

rufuspollock opened this issue Aug 18, 2020 · 14 comments
Assignees

Comments

@rufuspollock
Copy link
Member

rufuspollock commented Aug 18, 2020

A Data Dictionary (or Table Schema) provides information about the data in a resource such as the names, types and description of the fields.

https://tech.datopian.com/publish/#adding-a-schema-data-dictionary-for-a-resource

UX

For more, see https://tech.datopian.com/publish/#adding-a-schema-data-dictionary-for-a-resource

Horizontal (spreadsheet like)

Vertical layout

Acceptance

Working table schema editor as react component

UX is along lines of https://tech.datopian.com/publish/#adding-a-schema-data-dictionary-for-a-resource (but feel free to adapt/innovate)

Basic

  • Initialize with Data Resource including a Table Schema
    • Can assume sample data is already inlined in sample attribute as array of dicts
  • Can edit and save
  • Basic UI: Show fields/headers and then key attributes for table schema
    • title
    • description (markdown?)
    • type (string, int etc)
    • format (optional)
  • Show the sample of the data either with the headers or separately from the headers
    • Probably with the headers is easier

Infer

  • Given Data Resource without Table Schema auto infer table schema

Bonus / future

  • Show validation errors as you change types
  • (?) If no sample present auto generate the sample

Tasks

  • Research https://github.com/datopian/import-ui
  • Stub a Data Resource fixture (with Table Schema)
  • Implement the basic editor (not sure about order of tasks so choose what you neeed)
    • React component with key fields
    • Create a story using storybook for displaying the schema editor
    • Jest test

Analysis

JSON Schema editors

Research editors for json schema to see if we could reuse for our purposes ...

Design

Here's a rough code sketch of how this component could work:

<TableSchema
  // props for initialization
  // http://frictionlessdata.io/table-schema
  // https://specs.frictionlessdata.io/table-schema/
  schema= Frictionless (f11s) Table Schema object
  // sample of the data for displaying etc
  sample= [{ a: 1, b: 2}, {a: 5, b: 10}]
  />

Schema example for this case:

schema = {
  fields: [
    {
      name: 'a',
      type: 'string',
      description: 'column a is about X',
      format: '...'
    },
    {
      name: 'b',
      type: 'integer'
    }
  ]
}

Inside TableSchema component:

TableSchema.props.schema = make a copy of passed in schema
As you edit in UI the internal schema object is updated

LATER/FUTURE: For inference and having this work with data files ...

resource = open(HTML5 file object)

TableSchema(resource.schema, getSample(resource))

Use https://github.com/datopian/data.js for opening files ... (why? gives you all the metadata)

@rufuspollock rufuspollock changed the title [epic] Data Dictionary (table schema) for resources including inference, editing and saving [epic] Schema (data dictionary) for resources including inference, editing and saving Aug 18, 2020
@rufuspollock rufuspollock changed the title [epic] Schema (data dictionary) for resources including inference, editing and saving [epic] Schema (data dictionary) for resources w/ inference, editing and saving Aug 18, 2020
@rufuspollock rufuspollock changed the title [epic] Schema (data dictionary) for resources w/ inference, editing and saving [epic] Schema (data dictionary) for resources w/ infer, edit and save Aug 18, 2020
@rufuspollock rufuspollock changed the title [epic] Schema (data dictionary) for resources w/ infer, edit and save [epic] Resource Schema (data dictionary) editor w/ infer Aug 20, 2020
@mariorodeghiero
Copy link
Contributor

@abhishekgahlot @rufuspollock updates: I pushed this commit with the updates for to render a table with react-table.

Note: import-ui is using a different version of the react-table, if you need to check how the new version works, please check it in https://react-table.tanstack.com/docs/overview

@rufuspollock
Copy link
Member Author

@mariorodeghiero we should use the recent react-table. The next step here is to make the editor where you can edit the table schema info i think ...

mariorodeghiero added a commit that referenced this issue Aug 24, 2020
- create a new state of the schema
- add new css file
- render the schema on the table
- add input type text to be able to edit the state of the schema object
@mariorodeghiero
Copy link
Contributor

mariorodeghiero commented Aug 25, 2020

@anuveyatsu @rufuspollock I read and check the JSON schema editors available but I preferred to create my own and control the schema object via the state in react. The example is very simple yet but it works properly.

Online in https://datopian.github.io/datapub/?path=/story/components-tableschema--default

I think is important to decide one thing:

  • Does the state of the schema and onchange function and "onSave function" should be in the father of the component or in the TableSchema component?

@rufuspollock what is the title? Do we need to use?

Tomorrow I pretend to add select options to the user choose.

editor

@anuveyatsu
Copy link
Member

Hi @mariorodeghiero this is cool 👏

Are you referring to this spec here https://specs.frictionlessdata.io/table-schema/#descriptor ?

@rufuspollock
Copy link
Member Author

rufuspollock commented Aug 25, 2020

@mariorodeghiero great job. Note:

  • the types should come from the defined list that Table Schema has.
  • We also need to add title and that's it ...

I agree we don't need json schema editors here - just build our own given how simple this.

@mariorodeghiero
Copy link
Contributor

Hi @anuveyatsu I think we are up to date after our meeting today.

@rufuspollock yes, I will working to get this done and I will let you know.

mariorodeghiero added a commit that referenced this issue Aug 25, 2020
- add select options
- change the state of the options
- auto infer the type from the table schema
- add save schema button
@mariorodeghiero
Copy link
Contributor

@rufuspollock @anuveyatsu I push some changes for:

  • the types should come from the defined list that Table Schema has.
  • We also need to add title and that's it ...

Could please confirm if the types should be this below?

{
  "type": [
    "string",
    "number",
    "integer",
    "boolean",
    "object",
    "array",
    "null",
    "any"
  ]
}

Note: The button save only send a console.log with the current state of the table schema.

Screenshot from 2020-08-25 17-24-05

@anuveyatsu
Copy link
Member

@mariorodeghiero type options are listed below from the frictionlessdata spec:

  • string
  • number
  • integer
  • boolean
  • object
  • array
  • date
  • time
  • datetime
  • year
  • yearmonth
  • duration
  • geopoint
  • geojson
  • any

@mariorodeghiero
Copy link
Contributor

@anuveyatsu thanks. I will update the list of types and send a new PR.

@rufuspollock
Copy link
Member Author

@anuveyatsu @mariorodeghiero is this now complete?

@mariorodeghiero
Copy link
Contributor

@rufuspollock All things working only pending tests.
cc: @anuveyatsu

@rufuspollock
Copy link
Member Author

@mariorodeghiero do we have any tests for this? What would test look like? I thought we had tests for the React component ...

@mariorodeghiero
Copy link
Contributor

@rufuspollock We only created a simple test at the beginning of the datapub development.

@rufuspollock
Copy link
Member Author

FIXED. We can do more tests and that will be a separate issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants