New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Project Saving 2.0 #352

Closed
stevekrouse opened this Issue Apr 17, 2017 · 6 comments

Comments

Projects
1 participant
@stevekrouse
Owner

stevekrouse commented Apr 17, 2017

Our current schema:

code
--project-name
  --uid
  --revisionId
    --time
    --code
    --uid
user
--uid
  --createdDate
  --displayName
  --email

However, because revisions are stored under projects:

  • slow loading on large projects with many revisions
  • woofjs.com/analytics loads everything and is terribly slow

So a normalized schema would look like:

code-2
--project-name
  --uid
  --time
  --version-meta-data
    --revision-id
    --time
versions
  --version-id
    --uid
    --project-name
    --time
    --code

However, this also seems like a good place to add autosave #254, so that would look the same as above, except a current-version attribute on a project that contains the current code. On a new editing session, if the current-version is different from the last version, we save it as a new version. We then save every new edit to the current-version. Every 5 or 10 minutes, we preserve the current-version as a version. We also have a "save version" button.

code-2
--project-name
  --uid
  --time
  --current-version
    --code
    --time
  --version-meta-data
    --revision-id
    --time
versions
  --version-id
    --uid
    --project-name
    --time
    --code

This could be doing too much at once, but it seems like #307 collaborative, google-docs style editing is within our grasp here as well.

code-2
--project-name
  --uid
  --time
  --current-version
    --code
    --time
  --edit-meta-data
    --revision-id
    --time
edits
  --edit-id
    --uid
    --project-name
    --time
    --OT edit

However, we still need to figure out permissioning here. Some options:

  1. "Create a collaborative project" button to create new project which allows anyone with the link the edit that project at the same time. "Collaborative projects can be edited by multiple at the same time. Keep your project's name secret and only share it with people you trust. If you want to share your collaborative project more widely, create a non-collaborative project and copy and paste your collaborative project's code in there." If you ever made an edit on a collaborative project, it will show up in your list of projects under "Collaborative Projects"

  2. "Add collaborators to this project" button to add a collaborator by typing their username exactly (no search). We will probably also need a "remove collaborators" button as well. If you are added as a collaborator to a project, it will appear under your list of projects with the label "Collaborator." All projects you created will be labeled "Creator."

@stevekrouse

This comment has been minimized.

Show comment
Hide comment
@stevekrouse

stevekrouse Apr 17, 2017

Owner

I'm not sure how neccesary it is, but doing this (or parts of) #233 first might make collaborative permissioning easier

Owner

stevekrouse commented Apr 17, 2017

I'm not sure how neccesary it is, but doing this (or parts of) #233 first might make collaborative permissioning easier

@stevekrouse stevekrouse moved this from ToDo to Maybe ToDo in Steve's 5/27/17 sprint May 27, 2017

@stevekrouse stevekrouse moved this from Maybe ToDo to ToDo in Steve's 5/27/17 sprint May 28, 2017

@stevekrouse

This comment has been minimized.

Show comment
Hide comment
@stevekrouse

stevekrouse May 29, 2017

Owner

Here's my current plan. I'm going to leave our current database of projects alone. I'm going to create a new "table" for collaborative projects on woofjs.com/team. This will be a good testing group for OT. If it goes well, I can ETL our existing database of projects into /team by computing the OT transform between each subsequent revision.

Open questions:

  • Viewing old versions of your project. Can we let you scrub through the past like a movie? Is that too computationally or memory intensive?
  • Are we going to need to store a cached current version or will we constantly compute it from all the edits? (or will the library take care of this?)

New Schema

/team
  /uniqueProjectId
    /userId
    /createdTime
    /lastEditTime
    /currentCode
    /edits
     /edit1Id
     /edit2Id
/edits
  /editId
    /userId
    /uniqueProjectId
    /time
    /OT edit

I'm not realizing that when we merge this system with the regular non-collaborative projects, we'll need to switch things up a bit. If we only want the creator of a project to be able to edit it, we'll need to store the edits under the user's ID. So we'll 1) go to the project, 2) get edits for the project that are nested under that user's ID. (So another user would be able to add edits to that project under their own ID but it would be impossible to retrieve those edits through the UI.)

Owner

stevekrouse commented May 29, 2017

Here's my current plan. I'm going to leave our current database of projects alone. I'm going to create a new "table" for collaborative projects on woofjs.com/team. This will be a good testing group for OT. If it goes well, I can ETL our existing database of projects into /team by computing the OT transform between each subsequent revision.

Open questions:

  • Viewing old versions of your project. Can we let you scrub through the past like a movie? Is that too computationally or memory intensive?
  • Are we going to need to store a cached current version or will we constantly compute it from all the edits? (or will the library take care of this?)

New Schema

/team
  /uniqueProjectId
    /userId
    /createdTime
    /lastEditTime
    /currentCode
    /edits
     /edit1Id
     /edit2Id
/edits
  /editId
    /userId
    /uniqueProjectId
    /time
    /OT edit

I'm not realizing that when we merge this system with the regular non-collaborative projects, we'll need to switch things up a bit. If we only want the creator of a project to be able to edit it, we'll need to store the edits under the user's ID. So we'll 1) go to the project, 2) get edits for the project that are nested under that user's ID. (So another user would be able to add edits to that project under their own ID but it would be impossible to retrieve those edits through the UI.)

@stevekrouse

This comment has been minimized.

Show comment
Hide comment
@stevekrouse

stevekrouse Jun 2, 2017

Owner

Ok, new plan... I integrated with Firepad with like three lines of code to make collaborative editing work on woofjs.com/team #385. It seems to be working great and doesn't take up too much space. However, I'm not really sure how to process the data myself, so I'm not sure how to allow students to scroll back and forth in their history. We could do the stupid thing and just create versions on top of this but that feels silly -- I'll figure out the data structures eventually.

So my plan is to have /team data stay separate from the rest of the data. My new plan is to leave /code alone as the massive store of all of the past versions of every project. I will create a new place /project-no-versions that will store:

  • the current autosaved version
  • timestamp of the most recent autosave
  • the user who created the project
  • the list of all of the hashes of versions of the project

The hashes of all of the versions will be stored in the /code database as they are now. I will write a script to create the /project-no-versions store from the /code store basically by taking all of the "meta-data about projects." Going forward, projects will autosave every X changes or Y minutes to the /project-no-versions store and not touch the /code store unless you "save a version" or Z time/changes has past and we "auto save a version for you."

One problem I'm noticing... part of why we're doing this is making the data more digestible on woofjs.com/analytics. I really want a weekly active users number. I want to make a few different dashboards:

  1. lean growth metrics
  2. vanity metrics
  3. student and project database

However, I want to think about how to store the data about the number/timestamp of autosaves. For starters, I think it'd be ok to store it in /project-no-versions but eventually we might need a smarter place for that. Maybe in a firebase/analytics store? Maybe an event capture system like Google analytics or snowplow?

Owner

stevekrouse commented Jun 2, 2017

Ok, new plan... I integrated with Firepad with like three lines of code to make collaborative editing work on woofjs.com/team #385. It seems to be working great and doesn't take up too much space. However, I'm not really sure how to process the data myself, so I'm not sure how to allow students to scroll back and forth in their history. We could do the stupid thing and just create versions on top of this but that feels silly -- I'll figure out the data structures eventually.

So my plan is to have /team data stay separate from the rest of the data. My new plan is to leave /code alone as the massive store of all of the past versions of every project. I will create a new place /project-no-versions that will store:

  • the current autosaved version
  • timestamp of the most recent autosave
  • the user who created the project
  • the list of all of the hashes of versions of the project

The hashes of all of the versions will be stored in the /code database as they are now. I will write a script to create the /project-no-versions store from the /code store basically by taking all of the "meta-data about projects." Going forward, projects will autosave every X changes or Y minutes to the /project-no-versions store and not touch the /code store unless you "save a version" or Z time/changes has past and we "auto save a version for you."

One problem I'm noticing... part of why we're doing this is making the data more digestible on woofjs.com/analytics. I really want a weekly active users number. I want to make a few different dashboards:

  1. lean growth metrics
  2. vanity metrics
  3. student and project database

However, I want to think about how to store the data about the number/timestamp of autosaves. For starters, I think it'd be ok to store it in /project-no-versions but eventually we might need a smarter place for that. Maybe in a firebase/analytics store? Maybe an event capture system like Google analytics or snowplow?

@stevekrouse

This comment has been minimized.

Show comment
Hide comment
@stevekrouse

stevekrouse Jun 3, 2017

Owner

In theory, we don't need the list of all the hashes in the metadata table. We really just want that data in the analytics store somewhere so we can query it so we know how often people use that feature.

We can also employ some tricks to make this efficient, only pulling the version we need at a time. First we need to add indices a la https://stackoverflow.com/questions/35079061/can-i-get-the-nth-item-of-a-firebase-query and then use .startAt(version).limitTo(1).

Owner

stevekrouse commented Jun 3, 2017

In theory, we don't need the list of all the hashes in the metadata table. We really just want that data in the analytics store somewhere so we can query it so we know how often people use that feature.

We can also employ some tricks to make this efficient, only pulling the version we need at a time. First we need to add indices a la https://stackoverflow.com/questions/35079061/can-i-get-the-nth-item-of-a-firebase-query and then use .startAt(version).limitTo(1).

@stevekrouse

This comment has been minimized.

Show comment
Hide comment
@stevekrouse

stevekrouse Aug 27, 2017

Owner

Also, I want to consider a more SQL-like graphQL solution here, even though I'm 80% sure I'm sticking with firebase

https://news.ycombinator.com/item?id=13188774

Owner

stevekrouse commented Aug 27, 2017

Also, I want to consider a more SQL-like graphQL solution here, even though I'm 80% sure I'm sticking with firebase

https://news.ycombinator.com/item?id=13188774

@stevekrouse stevekrouse referenced this issue Sep 4, 2017

Open

Autosave #487

@stevekrouse

This comment has been minimized.

Show comment
Hide comment
@stevekrouse

stevekrouse Sep 4, 2017

Owner

Closed #481

Autosave is continued here: #487

Owner

stevekrouse commented Sep 4, 2017

Closed #481

Autosave is continued here: #487

@stevekrouse stevekrouse closed this Sep 4, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment