Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DRAFT] Workspace layer refactor #7657

Closed
wants to merge 52 commits into from
Closed
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
46c524c
wip
eanders-ms Nov 23, 2020
cc33cf2
wip
darzu Dec 1, 2020
010da76
show icon for cloud projects
darzu Dec 1, 2020
5f8085b
remove debug statements
darzu Dec 1, 2020
e15f586
investigative changes
darzu Dec 3, 2020
195b4e1
trying out cloudworkspace again
darzu Dec 4, 2020
20328c6
add browserdbworkspace.ts
darzu Dec 4, 2020
6909468
refactor browser workspace to build off more pure table db worksapce
darzu Dec 4, 2020
1c81516
error TOOO
darzu Dec 7, 2020
b19911d
debug logging
darzu Dec 7, 2020
c975d5d
create workspaces/ folder
darzu Dec 7, 2020
3ff57dd
refactor setupWorkspace into pure chooseWorkspace
darzu Dec 8, 2020
a5063a7
export
darzu Dec 8, 2020
423fe0a
cloud debug logging
darzu Dec 8, 2020
7c73231
debugging and type cleanup
darzu Dec 8, 2020
142b6c6
investigative comments
darzu Dec 8, 2020
b64be99
starting JointWorkspace
darzu Dec 8, 2020
631f200
toying with sync workspace
darzu Dec 9, 2020
c0fd88f
trying out joint workspace
darzu Dec 9, 2020
6d05fa5
experimenting
darzu Dec 9, 2020
98d30cb
unreachable helper
darzu Dec 9, 2020
c7bed33
header comments
darzu Dec 9, 2020
ffdcbad
debug logging
darzu Dec 9, 2020
0300ad1
rename
darzu Dec 9, 2020
7da5f13
taking a stab at synchronization
darzu Dec 9, 2020
dfbd3db
wip on sync
darzu Dec 10, 2020
a7523ad
trying cloud sync workspace again
darzu Dec 11, 2020
bc8a1a3
lots of work on cloud sync
darzu Dec 12, 2020
b8e8393
much better joint workspace
darzu Dec 13, 2020
f8201ce
fix build issue
darzu Dec 14, 2020
6939562
fix lint issues
darzu Dec 14, 2020
e6c3f9b
lots of debugging
darzu Dec 16, 2020
94605ea
debugging
darzu Dec 16, 2020
592c1ec
horrible debugging
darzu Dec 17, 2020
7971ff3
fix project transfer & debugging
darzu Dec 17, 2020
88717cf
fixining sync
darzu Dec 18, 2020
d711126
fix debug crash
darzu Dec 19, 2020
71941fd
debugging
darzu Dec 21, 2020
ae41778
big progress on race conditions & sanity
darzu Dec 24, 2020
be015c5
deprecate allscripts
darzu Dec 24, 2020
e83fd6e
investigative comments and todos
darzu Dec 28, 2020
8700481
reworked headers hash (formerly last-mod-time)
darzu Dec 29, 2020
49fb645
todo list and progress on syncAsync
darzu Dec 29, 2020
6b388a8
return headers[] from sync; finer grain virtual api inval
darzu Jan 2, 2021
e970d53
better names
darzu Jan 2, 2021
bb676aa
unify and fix up maybeSyncHeadersAsync, refreshHeadersSession and syn…
darzu Jan 3, 2021
a62252d
undo typo
darzu Jan 3, 2021
1d52319
debugging spurious saves
darzu Jan 4, 2021
a3155bd
first-save TODOs
darzu Jan 4, 2021
d70f46a
debugging empty saves
darzu Jan 4, 2021
0ea5813
fix issues with getTextAsync and lookup(); investigating double save …
darzu Jan 4, 2021
1e93dd1
handling 404s from the cloud
darzu Jan 4, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
15 changes: 10 additions & 5 deletions localtypings/projectheader.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@ declare namespace pxt.workspace {
tutorialCompleted?: pxt.tutorial.TutorialCompletionInfo;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the difference between "Header" and "InstallHeader" ?

// workspace guid of the extension under test
extensionUnderTest?: string;
cloudSync?: boolean; // Mark a header for syncing with a cloud provider
Copy link
Contributor Author

@darzu darzu Nov 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note:
was used to determine if a project was cloud synced (old Sam work?)
problem was it didn't distinguish account A or B

// id of cloud user who created this project
cloudUserId?: string;
}

export interface Header extends InstallHeader {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

header is meta info for a project

Expand All @@ -41,10 +42,14 @@ declare namespace pxt.workspace {
isDeleted: boolean; // mark whether or not a header has been deleted
saveId?: any; // used to determine whether a project has been edited while we're saving to cloud

// For cloud providers
blobId: string; // id of the cloud blob holding this script
blobVersion: string; // version of the cloud blob
blobCurrent: boolean; // has the current version of the script been pushed to cloud
// For cloud providers -- DEPRECATED
blobId_: string; // id of the cloud blob holding this script
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note that "project id" is assigned by client and its user ID + project ID. Note "id" field

blobVersion_: string; // version of the cloud blob
blobCurrent_: boolean; // has the current version of the script been pushed to cloud
darzu marked this conversation as resolved.
Show resolved Hide resolved

cloudVersion: string; // The cloud-assigned version number (e.g. etag)
// TODO @darzu: "cloudCurrent" seems very bad. This is a stateful notation and it is hard to reason about whether or not this is true.
cloudCurrent: boolean; // Has the current version of the project been pushed to cloud
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: do we need / how do we use cloudCurrent exactly?


// Used for Updating projects
backupRef?: string; // guid of backed-up project (present if an update was interrupted)
Expand Down
2 changes: 2 additions & 0 deletions localtypings/pxtpackage.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ declare namespace pxt {

type CodeCardType = "file" | "example" | "codeExample" | "tutorial" | "side" | "template" | "package" | "hw" | "forumUrl" | "forumExample" | "sharedExample";
type CodeCardEditorType = "blocks" | "js" | "py";
type CodeCardCloudState = "local" | "cloud";

interface Map<T> {
[index: string]: T;
Expand Down Expand Up @@ -159,6 +160,7 @@ declare namespace pxt {
cardType?: CodeCardType;
editor?: CodeCardEditorType;
otherActions?: CodeCardAction[];
cloudState?: CodeCardCloudState;

header?: string;

Expand Down
5 changes: 2 additions & 3 deletions pxteditor/editor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -215,12 +215,11 @@ namespace pxt.editor {
importExampleAsync(options: ExampleImportOptions): Promise<void>;
showScriptManager(): void;
importProjectDialog(): void;
cloudSync(): boolean;
cloudSignInDialog(): void;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cloudSignInDialog is unneeded because we have new dialog

cloudSignOut(): void;
removeProject(): void;
editText(): void;

hasCloudSync(): boolean;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

renamed from cloudSync to find usages

checks to see if we're signed in.


getPreferredEditor(): string;
saveAndCompile(): void;
updateHeaderName(name: string): void;
Expand Down
9 changes: 6 additions & 3 deletions pxteditor/workspace.ts
Original file line number Diff line number Diff line change
Expand Up @@ -54,9 +54,12 @@ namespace pxt.workspace {
id: U.guidGen(),
recentUse: modTime,
modificationTime: modTime,
blobId: null,
blobVersion: null,
blobCurrent: false,
blobId_: null,
blobVersion_: null,
blobCurrent_: false,
cloudUserId: null,
cloudCurrent: false,
cloudVersion: null,
isDeleted: false,
}
return header
Expand Down
74 changes: 13 additions & 61 deletions webapp/src/app.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ function setEditor(editor: ProjectView) {
}

export class ProjectView
extends data.Component<IAppProps, IAppState>
extends auth.Component<IAppProps, IAppState>
implements IProjectView {
editor: srceditor.Editor;
editorFile: pkg.File;
Expand Down Expand Up @@ -173,7 +173,6 @@ export class ProjectView
this.openDeviceSerial = this.openDeviceSerial.bind(this);
this.toggleGreenScreen = this.toggleGreenScreen.bind(this);
this.toggleSimulatorFullscreen = this.toggleSimulatorFullscreen.bind(this);
this.cloudSignInComplete = this.cloudSignInComplete.bind(this);
this.toggleSimulatorCollapse = this.toggleSimulatorCollapse.bind(this);
this.showKeymap = this.showKeymap.bind(this);
this.toggleKeymap = this.toggleKeymap.bind(this);
Expand Down Expand Up @@ -1335,7 +1334,10 @@ export class ProjectView
if (editorState.searchBar === undefined) editorState.searchBar = oldEditorState.searchBar;
}

if (!h.cloudSync && this.cloudSync()) h.cloudSync = true;
// If user is signed in, sync this project to the cloud.
if (this.hasCloudSync()) {
h.cloudUserId = this.getUser()?.id;
}

return compiler.newProjectAsync()
.then(() => h.backupRef ? workspace.restoreFromBackupAsync(h) : Promise.resolve())
Expand Down Expand Up @@ -2006,62 +2008,6 @@ export class ProjectView
})
}

///////////////////////////////////////////////////////////
//////////// Cloud ////////////
///////////////////////////////////////////////////////////

cloudSync() {
return this.hasSync();
}

cloudSignInDialog() {
const providers = cloudsync.providers();
if (providers.length == 0)
return;
if (providers.length == 1)
providers[0].loginAsync().then(() => {
this.cloudSignInComplete();
})
else {
// TODO: Revisit in new cloud sync
//this.signInDialog.show();
}
}

cloudSignOut() {
core.confirmAsync({
header: lf("Sign out"),
body: lf("You are signing out. Make sure that you commited all your changes, local projects will be deleted."),
agreeClass: "red",
agreeIcon: "sign out",
agreeLbl: lf("Sign out"),
}).then(r => {
if (r) {
const inEditor = !!this.state.header;
// Reset the cloud workspace
return workspace.resetCloudAsync()
.then(() => {
if (inEditor) {
this.openHome();
}
if (this.home) {
this.home.forceUpdate();
}
})
}
return Promise.resolve();
});
}

cloudSignInComplete() {
pxt.log('cloud sign in complete');
initLogin();
cloudsync.syncAsync()
.then(() => {
this.forceUpdate();
}).done();
}

///////////////////////////////////////////////////////////
//////////// Home /////////////
///////////////////////////////////////////////////////////
Expand Down Expand Up @@ -2251,7 +2197,7 @@ export class ProjectView
pubCurrent: false,
target: pxt.appTarget.id,
targetVersion: pxt.appTarget.versions.target,
cloudSync: this.cloudSync(),
cloudUserId: this.getUser()?.id,
temporary: options.temporary,
tutorial: options.tutorial,
extensionUnderTest: options.extensionUnderTest
Expand Down Expand Up @@ -3090,6 +3036,10 @@ export class ProjectView
}
}

hasCloudSync() {
return this.isLoggedIn();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: maybe once we go to AAD we might not be able to store data ourselves and we might need OneDrive? Probably worth adding a TODO comment

}

showScriptManager() {
this.scriptManagerDialog.show();
}
Expand Down Expand Up @@ -4545,7 +4495,9 @@ document.addEventListener("DOMContentLoaded", () => {
else if (isSandbox) workspace.setupWorkspace("mem");
else if (pxt.winrt.isWinRT()) workspace.setupWorkspace("uwp");
else if (pxt.BrowserUtils.isIpcRenderer()) workspace.setupWorkspace("idb");
else if (pxt.BrowserUtils.isLocalHost() || pxt.BrowserUtils.isPxtElectron()) workspace.setupWorkspace("fs");
// TODO @darzu: uncomment. this disables filesystem workspace
//else if (pxt.BrowserUtils.isLocalHost() || pxt.BrowserUtils.isPxtElectron()) workspace.setupWorkspace("fs");
Copy link
Contributor Author

@darzu darzu Nov 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO uncomment

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this disabled the filesystem workspace

else workspace.setupWorkspace("browser");
Promise.resolve()
.then(async () => {
const href = window.location.href;
Expand Down
10 changes: 8 additions & 2 deletions webapp/src/auth.ts
Original file line number Diff line number Diff line change
Expand Up @@ -465,12 +465,18 @@ export async function initialUserPreferences(): Promise<UserPreferences | undefi
return initialUserPreferences_;
}

function loggedInSync(): boolean {
export function loggedInSync(): boolean {
if (!hasIdentity()) { return false; }
const state = getState();
return !!state.profile?.id;
}

export function user(): UserProfile {
if (!hasIdentity()) { return null; }
const state = getState();
return { ...state.profile };
}

async function fetchUserAsync(): Promise<UserProfile | undefined> {
const state = getState();

Expand Down Expand Up @@ -637,5 +643,5 @@ data.mountVirtualApi(MODULE, { getSync: authApiHandler });


// ClouddWorkspace must be included after we mount our virtual APIs.
import * as cloudWorkspace from "./cloudworkspace";
import * as cloudWorkspace from "./cloud";
cloudWorkspace.init();
62 changes: 62 additions & 0 deletions webapp/src/browserdbworkspace.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
import * as db from "./db";

type Header = pxt.workspace.Header;
type ScriptText = pxt.workspace.ScriptText;

type TextDbEntry = {
id: string,
files?: ScriptText,
_rev: any
}

export interface BrowserDbWorkspaceProvider extends pxt.workspace.WorkspaceProvider {
prefix: string;
}

export function createBrowserDbWorkspace(namespace: string): BrowserDbWorkspaceProvider {
const prefix = namespace ? namespace + "-" : ""
const headerDb = new db.Table(`${prefix}header`);
const textDb = new db.Table(`${prefix}text`);

async function listAsync(): Promise<pxt.workspace.Header[]> {
return headerDb.getAllAsync()
}
async function getAsync(h: Header): Promise<pxt.workspace.File> {
const resp: TextDbEntry = await textDb.getAsync(h.id)
return {
header: h,
text: resp.files,
version: resp._rev
}
}
async function setAsync(h: Header, prevVer: any, text?: ScriptText): Promise<string> {
const textEnt: TextDbEntry = {
id: h.id,
files: text,
_rev: prevVer
}
const retrev = await textDb.setAsync(textEnt)
const rev = await headerDb.setAsync(h)
h._rev = rev
return retrev
}
async function deleteAsync(h: Header, prevVer: any): Promise<void> {
await headerDb.deleteAsync(h)
await textDb.deleteAsync({ id: h.id, _rev: h._rev })
}
async function resetAsync() {
// workspace.resetAsync already clears all tables
// TODO @darzu: I don't like that worksapce reset does that....
return Promise.resolve();
}

const provider: BrowserDbWorkspaceProvider = {
prefix,
getAsync,
setAsync,
deleteAsync,
listAsync,
resetAsync,
}
return provider;
}