New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Add a Data Guide (or guides) #4014

Open
justinbmeyer opened this Issue Mar 7, 2018 · 11 comments

Comments

Projects
None yet
5 participants
@justinbmeyer
Copy link
Contributor

justinbmeyer commented Mar 7, 2018

TLDR: A guide to teach people how to use CanJS's data layer (can-connect, can-query-logic, etc). This will be created after can-query-logic is finished.

This was discussed on a recent live stream (28:37) and previously here (32:34) and here (22:43).

YOU CAN HELP BY:

  • Adding in the comments the use cases you've encountered that you would like to see covered.
  • What type of service layer are you connecting to? What does it look like to query a list of items? Does your server handle relationships?

Motivation

CanJS has a lot of powerful tools around data modeling - real-time, caching, "instance awareness".
But these tools are underutilized for probably two reasons:

  1. Lack of awareness of how they work, and what they do.
  2. Configuration difficulty. Our tools require a high degree of conformity to be used easily. CanJS expects your data to look a certain way, your services to work a certain way, etc. While these tools do make customization possible, it's hard to figure out.

Solution

I propose a series of guides for working with CanJS's data layer:

First:

  • An overview of building a data-model layer against a "perfect" data service. The data-layer will be added piece-by-piece up to including everything in superMap. The goal will be to teach people the concepts and pieces in an ideal environment.

Then a bunch of micro guides on dealing with less than ideal environment:

  • Alternate query params
  • Alternate REST services (urls are wrong, data is wrong)

Then some micro guides on specific areas

  • Relationships
  • Sessions

Outline

Rough Content

Purpose

The purpose of CanJS's data modeling packages is to make it
easy to connect your application to a backend data source.

While fetch, [can-ajax], and XMLHTTPRequest can be used directly in a CanJS application,
using CanJS's data modeling tools can solve some difficult problems with little configuration:

  • Provide a standard interface for retrieving, creating, updating, and deleting data.
  • Convert raw data from the server to typed data, with methods and special property behaviors.
  • Caching
  • Real time updates of instances and lists
  • Prevent multiple instances of a given id or multiple lists of a given set from being created.
  • Handle relationships between data.

When to use

  • You want some of the benefits above.
  • when you can have lots of consistency on the server

Core Concepts

  • Types
  • Queries
  • Data Interface
  • Instance Interface
  • Behaviors

Types

Todo is a map-type, comprised of other types: Boolean, Number etc.

Serialized vs unserialized.

Queries

Represent sets of data. Can make use of the underlying type information.

How can {gt: 5} be made to generically work with String and Number types?

Data Interface

CRUD connection. All data involved is "serialized".

Instance Interface

Methods added to a type so you can CRUD remote representations of it.

Behaviors

Add or overwrite data and instance interface methods.

Use Cases

real-time

caching strategies

connecting to services for "normalized" sql-like databases (Bitballs type apps)

connecting to services for document-like databases (Bitcentive type apps)

how to handle the session

Singleton sessions like Bitcentive

  • Session.current - see if there is a current session. Observable.
  • new Session().save() - Create a new session.
  • Session.current.destroy() - Logout from anywhere.

how to handle relationships

Game.get({
    id: this.gameId,
    withRelated: ["stats",
        "homeTeam.player1",
        "homeTeam.player2",
        "homeTeam.player3",
        "homeTeam.player4",
        "awayTeam.player1",
        "awayTeam.player2",
        "awayTeam.player3",
        "awayTeam.player4"
    ]
});

Make sure that the list of stats will know who it belongs to:

stats: {
		Type: Stat.List,
		set: function(stats){
			if (stats) {
				stats[Stat.connection.listQueryProp] = {filter: {gameId: this.id }};
			}

			return stats;
		}
	},

Refs should make this easier.

  • There's a jsonapi version of this.

@chasenlehara chasenlehara changed the title Proposal: Data Guide Proposal: Add a Data Guide Mar 9, 2018

@justinbmeyer justinbmeyer changed the title Proposal: Add a Data Guide Proposal: Add a Data Guide (or guides) Mar 9, 2018

@mikemitchel

This comment has been minimized.

Copy link
Contributor

mikemitchel commented Nov 28, 2018

I voted for this mostly for this reason

CanJS has a lot of powerful tools around data modeling - real-time, caching, "instance awareness".
But these tools are underutilized for probably two reasons:

Lack of awareness of how they work, and what they do.
@chasenlehara

This comment has been minimized.

Copy link
Member

chasenlehara commented Nov 28, 2018

A couple of things I’ve seen in APIs that we should probably address:

  1. An endpoint that returns different data types, e.g. one API call that returns { data: { projects: [ ... ], todos: [ ... ] } } (how to make this call, how to manage caching, how to model the data, etc.)

  2. The opposite of 1, where there are different endpoints for slightly different types of data (e.g. /api/completed and /api/incomplete endpoints that return todos that have slightly different properties); how should you get all todos, how to manage caching, how to model data, etc.

@Lighttree

This comment has been minimized.

Copy link

Lighttree commented Nov 30, 2018

I think it makes sense to show how to work with Request Headers in CanJS (I assume this requires can-connect) and maybe other low level request manipulation.

Real life case is "Token based" authentication. For example JWT authentication is very popular nowdays, and in order to implement it, you have to add Bearer token to request header, and build flow around it. Check its expiration, try to get new token if expired, re-try original request with new token attached.

Internally we implemented this by behaviors, but since it looks like trivial scenario it makes sense to show how to do something like this.

@justinbmeyer

This comment has been minimized.

Copy link
Contributor

justinbmeyer commented Nov 30, 2018

@Lighttree, great suggestion! Anyway you can share some of the code for that behavior?

@justinbmeyer

This comment has been minimized.

Copy link
Contributor

justinbmeyer commented Nov 30, 2018

One thing I've been thinking about, is how to include the session information with the query ...

  • especially when the session impacts the query
/api/todos
-> unauthorized 

/api/todos JWT
A -> unauthorized 
B -> [ todos for that user ] SIMILAR TO /api/todos?filter[createdBy]=5
   Todo.getList({})

   {name: "should not be in your list", createdBy: 6, assignedTo: 5} -> would get added to a list!!
Todo.getList({})
  • What should be passed with the query?
    Todo.getList({token: XYZ, filter:{}})
    • token is not going to be on your data response ....
  • What could be read from some global state?
    getListData(){
      LOOKUP TOKEN 
      ADD TOKEN TO HEADERS
      return requestPromise
    }
@Lighttree

This comment has been minimized.

Copy link

Lighttree commented Nov 30, 2018

I'll share more info on Monday since I don't have access to code right now.

Main issues that we faced during implementation is that we weren't able to access 'request headers' and "before send" logic of original request. So additionaly to behavior that modify request we had to use custom ajax wrapper to met requirements.

It might be that today there is better way to do this. Anyway I'll return with more technical details on Monday :)

@eben-roux

This comment has been minimized.

Copy link
Contributor

eben-roux commented Dec 10, 2018

I had a really hard time trying to use can-connect that last time I had a look at it. I ended up rolling a rather simple interaction mechanism. This along with my shuttle-access for identity & access control gets me to where I need to be.

Any mechanism I use should be flexible enough to allow one to add any behaviour (if required) for various "events" within the interaction pipeline. If I need to add request headers then it should be a rather simple affair. In most cases dropping down to the relevant level does actually help but it would be nice to keep it in the can-connect component for the 80% cases.

For security I simply add a header like so:

$.ajaxPrefilter(function (options, originalOptions) {
    options.beforeSend = function (xhr) {
        if (access.token) {
            xhr.setRequestHeader('access-sessiontoken', access.token);
        }

        if (originalOptions.beforeSend) {
            originalOptions.beforeSend(xhr);
        }
    };
});

I don't think session management should be baked in. It should, however, be possible to add some implementation that works through a common interface. These things can be tricky so if it isn't going to help much then it could also be excluded.

For me the main focus should be the data interaction. How do I go about retrieving the data into the relevant DefineMap or DefineList structures? I also do not want to set up a massive can-connect configuration for each endpoint. There should be something that covers most cases and options that could override some behaviour; in some cases a totally different can-connect may be required but that should not happen all-too-often.

I would still suggest that each of the interaction pipelines have a predefined set of steps (events) it passes through and that one could attach an observer to any of those events to enrich the state. This roughly follows a pipes-and-filters mechanism.

Only base-bones functionality that is truly common should be baked-in and any other mechanisms or behaviours should be added explicitly.

@justinbmeyer

This comment has been minimized.

Copy link
Contributor

justinbmeyer commented Dec 11, 2018

how to handle the session

CRUD session JWT & Cookies

Pass the session (JWT)

Todo.getList({})

behavior = function(){

   obj = dataUrl(base);

   return Object.extend ( Object.create(base), obj )
}
// this.getData 
// base.getData
var connection = [base, dataUrl, constructor].reduce( (last, behavior) => behavior(last), {
  Map: Session
})
Object.extend(connection, {
  getData(){

  }
})

var connection = base()
connection = dataUrl(connection);
connection = constructor(connection)
connection = Object.create(connection, {
  getData(){ 

  }
})

// session.js 
restModel([dataUrl,{getData(){}}],{
  Map: Session
},{
  getData(){

  },
  destroyData(){

  },
  
})
import {Session}


Todo.connection = connect([
   Session.connection.addAsRequestHeader
],{
  
})

  • JWT & cookies

  • Singleton sessions like Bitcentive

    • Session.current - see if there is a current session. Observable.
    • new Session().save() - Create a new session.
    • Session.current.destroy() - Logout from anywhere.
Session.get()
Todo.getList()
  1. Wait for the session to be available
  2. Add session info
// wait for session to be established ... info before requesting stuff ...
beforeSend(xhr){
  return Session.currentPromise.then(function(session){
    
  });
},
url: "/api/todos"
  1. CRUD Session
  2. Singleton stuff
  3. Add session info on other requests

Bitballs

Session = DefineMap.extend({
});

import connectCanSession from "can-connect/can/session/session";

connect([restModelsBehaviors..., connectCanSession], {
  Map: Session
  // NO LIST IS OK ... 
  url: "/api/session",
  // JWT
  url: {
    getData(){ ... },
    createData() { ... }
  }
});

Session.current // First time it's read ... undefined, but kicks off a `Session.get()`

new Session({ ... }).save() //-> creates

Session.current.destroy() //-> delete, set Session.current to undefined

Session.currentPromise //-> resolved or rejected with the `Session.get()`


import Session from "./session"

Player = DefineMap.extend({ ... })

realtimeRestModel({
  Map: Player,
  url: {
    resource: "/api/players",
    beforeSend(xhr){
       return Session.currentPromise.then(function(session){
          xhr.setRequestHeader("AUTH",session.token);
       });
    }
  }
});
  • can-ajax work with a promise-based beforeSend - 3-4hrs (make beforeSend actually before send)
  • data-url beforeSend configuration ... document that - 3-4 hrs
  • can-connect/can/session/session .. document that - 2-3 days
    • fix can-connect/can/map needing a List type
  • write the guide :-)
@justinbmeyer

This comment has been minimized.

Copy link
Contributor

justinbmeyer commented Dec 11, 2018

Behaviors overview

  • hook vs MIXIE

  • getData from constructor's perspective ... is a plain hook .. calls it like this.getData()

  • getData from "combineRequest" "fall-through-cache" ... getData "MIXIE".

  • parseListData - hook

GOAL:

  • Know the existing behaviors, how a behavior works, and how they can be composed together.

.getList({})

.save()

  1. can/map - connection.save()

  2. can/map-.isSaving()` MIXIE

  3. constructor/store - IN => Makes sure instance is in the store

  4. can-connect/constructor/constructor IN => _CID store, CALLS .createData ... (OUT)

  5. data/callbacks MIXIE --- calls base

  6. real-time -> calls the base, but make sure promise doesn't resolve until any prior saves resolved

  7. can-connect/data/parse/parse -> calls base

  8. data-url makes the request

  9. can-connect/data/parse/parse -> PARSES DATA

  10. data/callbacks Calls .createdData

    1. real-time -> adds stuff to stores ... Calls createdInstance.
      1. constructor/callbacks-once -> Makes sure that we are only seeing created once. Calls base
      2. can-connect/can/map/map -> Dispatch "created" events
  11. callbacks-once

  • base

@justinbmeyer

This comment has been minimized.

Copy link
Contributor

justinbmeyer commented Dec 11, 2018

Debugging

  • .log()
  • See proto-chain
@Lighttree

This comment has been minimized.

Copy link

Lighttree commented Dec 12, 2018

Hi, sorry it took so long, to return with some details..

Basically our solution includes 2 parts: custom behavior and "customizable" jQuery ajax.

Behavior:
Its role to append token to every request where this behavior used. If token is expired it should renew it by sending another request to endpoint that generates them. (tokens are short-living and saved in sessionStorage to use them for request during life-time)

After this token generated it should append new token to request and re-try original request. If it succeed - ok. If generation of token fails - we redirect user to login.

import connect from 'can-connect';
import $ from 'jquery';

// This is endpoint that generates new tokens...
import JWTAuthToken from './JWTAuthToken.model';

const methods = ['getListData', 'getData', 'createData', 'updateData', 'destroyData'];

let setHeaders = function (xhr) {
    if (this) {
        xhr.setRequestHeader('Authorization', `Bearer ${this.token}`);
    }
};

const getToken = function (isRefreshNeeded) {
    let isJwtTokenExpired;
    let storedData = {};

    isJwtTokenExpired = ((new Date()).getTime() - /* token expiration date */) > 0;

    if (isRefreshNeeded || isJwtTokenExpired) {
        sessionStorage.removeItem('token');
    }

    if (!sessionStorage.getItem('token')) {
        return JWTAuthToken.get({}).then((data) => {
            let time = (new Date()).getTime();
            sessionStorage.setItem('token', data.token);
            return data;
        });
    }

    storedData.token = sessionStorage.getItem('token');

    return $.Deferred().resolve(storedData);
};

const JWTBehavior = connect.behavior('JWTBehavior', (baseConnection) => {
    const behavior = {};

    for (let i = 0; i < methods.length; i++) {
        behavior[methods[i]] = function (params) {
            let prom = $.Deferred();
            getToken().then((data) => {
                let self = this;
                let ajaxConf = {
                    tryCount: 0,
                    retryLimit: 1
                };
                ajaxConf.beforeSend = setHeaders.bind(data);

                ajaxConf.success = function (successData) {
                    prom.resolve(successData);
                };

                ajaxConf.error = function (xhr) {
                    this.tryCount++;
                    if (this.tryCount <= this.retryLimit && xhr.status === 401) {
                        getToken(true).then((secondResponse) => {
                            this.beforeSend = setHeaders.bind(secondResponse);
                            self.ajax(this);
                        }).catch(/* do something */);
                    } else {
                        prom.reject(xhr);
                    }
                };
                
                // This is possible due to custom ajax provided.
                this.ajax.setOptions(ajaxConf);

                baseConnection[methods[i]].call(this, params);
            }).catch(/* do something */);
            return prom;
        };
    }

    return behavior;
});

export default JWTBehavior;

After you can use such behavior in models:

import DefineMap from 'can-define/map/map';
import DefineList from 'can-define/list/list';
import connect from 'can-connect';
import url from 'can-connect/data/url/url';
import constructor from 'can-connect/constructor/constructor';
import canMap from 'can-connect/can/map/map';
import ajaxConfigurable from 'canjs-auth/ajaxConfigurable';
import JWTAuthBehavior from 'canjs-auth/JWTAuth';

const Entity = DefineMap.extend({ seal: false }, {
  // fields
});

Entity.List = DefineList.extend({
    '#': Entity
});

Entity.connection = connect([
    url,
    JWTAuthBehavior,
    constructor,
    canMap
], {
    url: 'secured/endpoint/entity',
    ajax: ajaxConfigurable(),
    Map: Entity,
    List: Entity.List
});

export default Entity;

This is done for Can@3 and maybe for Can@5 it can be done way more better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment