Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Lanes and Snaps features #2754

Merged
merged 183 commits into from Jul 17, 2020
Merged

Implement Lanes and Snaps features #2754

merged 183 commits into from Jul 17, 2020

Conversation

davidfirst
Copy link
Member

@davidfirst davidfirst commented Jun 18, 2020

See more details about these two features here: Lanes. Snaps.
Note that for now, these features are disabled by a feature toggle "lanes". Our recommendation is not to use it yet.

The following describes the final implementation, which differs from the specification above.

Synopsis

  • create a snap: bit snap (synopsis similar to the bit tag).
  • create a new lane: bit switch <name> --create
  • list lanes: bit lane.
  • switch between lanes: bit switch <name>
  • merge lanes: bit merge --lane.
  • show lane details: bit lane <name>
  • track local lane to a remote lane: bit switch --as
  • rename a lane bit switch --rename
  • remove a lane bit remove --lane.
  • fetch lane objects (without the components): bit fetch --lanes.
  • import a lane: bit switch <name> --remote.
  • export a lane: bit export <name> --lanes.
  • export current lane: bit export.
  • (internal) cat lane object: bit cat-lane <name>.

Remaining tasks

  • Remove components from a remote lane.
  • Bit import with no args. When bitmap has a remote lane, should import the lane.
  • Tag dependencies to include them in a lane.
  • Test bit-fetch components
  • Rename lanes.
  • "Bit diff" between lanes
  • Fix performance issue. Now it fetches all parents every time. It doesn't make sense.
  • Fix performance issue. Now it traverses all parents to see if a hash in there.

Important points

Performance consideration

Currently, if it imports with no-deps, it doesn't ask for parents, if it imports with deps it imports with all parents. It is originated from src/api/scope/lib/fetch.ts: const collectParents = !noDependencies;. We need to make a decision here.

Model changes

  • Version.parents. This is added only if the lanes feature is enabled. (it doesn't change the hash).
  • Component.head This is added only if the lanes feature is enabled. (it doesn't change the hash).

Hash

  • Snap's hash is generated by a UUID and then converted into sha1. sha1(v4()).
  • Lane's hash is generated by a UUID and then converted into sha1. sha1(v4()).
  • Tag's hash stays the same. Generated by the Version.id().

Lane Data

Lane data, for the most cases is a map of component-id:snap-head, in other words, it saves per component the head snap. There are 3 different places where we store such data for different purposes.

  • lane-object "Lane" is saved in the scope .bit/objects (.bit can be .git/bit locally), it has a unique hash and contains a map of components and their heads. This object exists on both local and remote scopes. Its main purpose is to sync lane-data between scopes. See Lane class (in scope/model) for the implementation details.
  • workspace-lane (component and their heads) is saved in .bit/workspace/lanes/lane-name. Remote scopes don't have this data. Its main purpose is to store data that exists only locally and does not exist on the remote. See WorkspaceLane class for the implementation details.
  • remote-lane (component and their heads) is saved in .bit/refs/remote/remote-name/lane-name. These refs files are saved in both, local and remote scopes. Its main purpose is to keep track where the remote-heads are per lane. See RemoteLanes class for the implementation details.

More places that stores lanes related data:

  • the currently checked out lane is saved in the scope.json file. (e.g. { lanes: { current: "lane-a", tracking: "remote/lane-a" } }).
  • When switching to a remote lane, the .bitmap is updated as well with the remote-name, so then, when cloning the project, it's possible to fetch the remote lane data.

Summary of when/what lanes data is saved per command:

  • bit switch --create: creates a new lane-object.
  • bit snap: adds an entry to the lane-object and to the workspace-lane.
  • bit export: 1) deletes all records from workspace-lane (as they're in sync with the remote). 2) pushes the lane-object to the remote. On the remote, the lane-object is created or updated/merged.
  • bit import/fetch: 1) creates/updates lane-object in the scope. 2) creates/updates remote-lane. the remote-lane is updated also for master.

Merge during import

  • When the remote is ahead, it's easy, just update the head to the remote-head.
  • When the local and remote have diverged, it's more complex. We need to create a snap-merge that has two parents, one point to the local head and one point to the remote head.
  • On bit import --objects (or bit fetch) we don't merge. The remote head is saved in the .bit/refs/remote dir. Then, bit status shows that these components are merge-pending.
  • On bit import <id> we do the merge.
  • in case the merge wasn't done and the user is trying to export, the remote blocks is as it finds out that its head doesn't exist on the incoming component.
  • in case the snap-merge failed due to a conflict, it saves the conflict and heads data into .bit/unmerged.json file.

Useful APIs

  • bit-map getAllIdsAvailableOnLane() filters the currently checked out lane.

@davidfirst davidfirst merged commit 78ba57c into master Jul 17, 2020
@davidfirst davidfirst deleted the feature/snap-merge branch July 17, 2020 00:53
@davidfirst davidfirst changed the title Merge master into Lanes branch Implement Lanes and Snaps features Jul 17, 2020
odedre added a commit that referenced this pull request Jul 19, 2020
commit c064f84
Author: Ran Mizrahi <ranm8@users.noreply.github.com>
Date:   Sun Jul 19 12:55:12 2020 +0300

    Harmony/scope (#2861)

    * component history working

    * builder api working

    * fixed component model when for new components

    * prettier2 execution

    * made workspace work as UI root and refactored few bad dependency relationships (e.g. bundler) to the workspace

    * scope is rendering

    * scope sidebar is rendering

    * workspace and scope are now both fetching from component host

    * lint errors

commit 030eeb7
Author: David First <davidnet@gmail.com>
Date:   Sat Jul 18 22:41:04 2020 -0400

    refactor Compiler extension and fix "main" package.json prop during link process (#2863)

    * refactor compiler extesnsion - remove dead code, change render() to report()

    * rename src/extensions/compiler/compiler.cmd.tsx to src/extensions/compiler/compiler.cmd.ts

    * restructure the "compile" class, extract the new compiler code to a new class CompileComponent

    * fix "main" prop of package.json written for author to have the correct file given by the extensions

    * rename src/extensions/compiler/compile.ts to src/extensions/compiler/workspace-compiler.ts

commit 09a5fec
Author: David First <davidnet@gmail.com>
Date:   Fri Jul 17 23:49:12 2020 -0400

    change compiler api. compileFile => transpileFile, compileOnCapsules => build (#2860)

commit 6c882b6
Author: David First <davidnet@gmail.com>
Date:   Fri Jul 17 20:52:41 2020 -0400

    fix Capsule.component to be Component and not ConsumerComponent (#2858)

    * fix Capsule.component to be Component and not ConsumerComponent

    * move capsule commands from isolate-ext to workspace-ext

commit caeddeb
Author: Ran Mizrahi <ranm8@users.noreply.github.com>
Date:   Sat Jul 18 03:47:44 2020 +0300

    Harmony/builder api (#2854)

    * component history working

    * builder api working

    * fixed component model when for new components

    * prettier2 execution

    Co-authored-by: Ran Mizrahi <ran@bit.dev>

commit 0043035
Author: David First <davidnet@gmail.com>
Date:   Fri Jul 17 20:07:14 2020 -0400

    add extra-data to Analytics to indicate whether the workspace is harmony (#2859)

commit 6bdc653
Author: David First <davidnet@gmail.com>
Date:   Fri Jul 17 16:01:17 2020 -0400

    change publisher to work with workspace only, remove createNetworkFromScope method (#2857)

commit 5bf6535
Author: David First <davidnet@gmail.com>
Date:   Fri Jul 17 15:20:03 2020 -0400

    fix "npm run format" to prettify also .js and .css extensions, similar to the lint-staged (#2856)

commit e403658
Author: David First <davidnet@gmail.com>
Date:   Fri Jul 17 15:18:54 2020 -0400

    rename compiler extension from "compile" to "@teambit/compiler" (#2855)

    also, change the extension to the new format (with a class).

commit 7138e50
Author: David First <davidnet@gmail.com>
Date:   Fri Jul 17 11:12:33 2020 -0400

    upgrade prettier from 1.x to 2.x and re-format all files (#2852)

commit 78ba57c
Author: David First <davidnet@gmail.com>
Date:   Thu Jul 16 20:53:10 2020 -0400

    implement "lanes" (#1986) and "snaps" (#1985) features (#2754)

    * implement the "lanes" feature, similar to the description in #1986.

    * implement the "snaps" feature, similar to the description in #1985.

    * disconnect the hash calculation of a Version object from its path, it helps to be flexible with schema changes and avoid HashMismatch exceptions

    * snap and tag are working in conjunction

    * snap - have head snap as Ref in memory

    * fix modified status after snap

    * change parent hash prop if version hash is changed

    * change parent prop to be an array of parents

    * fix snap head upon untag

    * fix untag for snaps

    * provide a more descriptive error message when a hash is missing

    * avoid fetching all versions when not needed

    * fix bug of second tag after snap

    * fix bit-log to show snaps. support diverged snaps history. fix import and export of snaps. write remote head to a new file at refs/remote

    * implement import --object when the local head is different than the remote head

    * (wip) started working on bit lane command

    * revert the implementation of saving different "Component" objects per lanes (done by dd20bb1)

    * implement Lane object (inherit BitObject)

    * add basic validation for "lane" object, introduce "bit cat-lane" command, fix "bit status" when snapping on master then on a new lane

    *  remove scope-name from lane-id

    * fix bit diff to work with snaps

    * bit checkout is working for Snaps

    * add tests for bit merge with snaps

    * recognize diverged components, suggest to merge them in bit status and throw error on bit import in case --merge was not entered

    * fix status of merge-pending to show the number of snaps existing on local/remote

    * add the common snap before diverge data into the diverge results

    * implement a snap-merge of diverged components

    * fix bit-status to show an unresolved component as such, implement bit merge --resolve

    * implement merge --abort

    * prevent tag/snap/checkout/merge a component with conflicts, enable untag/remove

    * basic lane export is working

    * write exported components heads into refs/remote

    * started implementing "bit import" lanes

    * move current-local-name from .bitmap to scope.json

    * basic functionality of switching between master to a lane is working

    * add test for "bit diff" when on a lane, fix "bit status" when snapped on a lane, refactor "diverge" data

    * Started working on "bit merge" for lanes. fix tag & checkout lane for multiple components

    * fix tests, fix exporting from master to write the snaps into the remote ref

    * basic "bit merge" of lanes is working

    * fix merging from a remote lane into master

    * start working on noSnap flag

    * consolidating merge-lanes with snap-merge logic. add merge-snaps output

    * implement --no-snap and --message flags of bit-merge

    * add existingOnWorkspaceOnly flag for bit-merge

    * add e2e-test and output when using --existing flag for bit-merge

    * implement bit lane --merge and --not-merged flags

    * add a flag "new-lane-name" for bit-checkout

    * introduce new flags for "bit import": --new-lane-name and --checkout

    * copy components data from origin lane to a newly created lane (except master)

    * introduce "bit switch" to switch/create/import lanes

    * introduce "bit fetch" command to fetch remote objects

    * improve lane-id

    * bit fetch is working, change tests to work with bit-switch and bit-fetch

    * basic case of switching from a lane back to master is working

    * basic case of importing a component when checked out to a lane

    * change bit-lane to function only as list/show not as "add". change the output according to the new specs

    * started working on workspace-lanes

    * implement workspace-lane object to save the current (checked out) versions per local lane

    * export with no ids when checked out to a lane, exports the lane

    * change the export output to indicate that lanes were exported

    * separate getAllBitIds and getAllIdsAvailableOnLane methods to not use the same cache

    * implement remove local lanes

    * enable fetching lane list from a remote

    * when user is checked out to a lane, "bit remove" removes the component from the lane only

    * enable removing remote lanes

    * block tagging when on a lane for now. we might change it later

    * implement auto-snap

    * skip a test about fetching dependencies parents for now, it will be back once the backward compatibility is implemented

    * implement auto-snap on lanes

    * fix ModelComponent.latest() to always work with the current lane

    * skip an e2e-test for now until backward compatibility is implemented

    * support feature-toggle mechanism

    * backward compatibility: do not save snaps.head and component parents when lane feature is disabled

    * backward compatibility - fix import from old clients

    * throw errors when trying to use lanes/snaps and the feature is disabled

    * re-enable some tests now that backward compatibility is supported

    * extract create-lane functionality to a new file

    * extract export-lanes functionality to a new file

    * refactoring. make sure fetch api gets bit-ids or remote-lane-ids. (remove the hack of converting lane-id into bit-id)

    * clean some unneeded flags on checkout command

    * fix MergeConflictOnRemote to be backward compatible

    * refactoring and mark lane flags as experimental

    * remove some dups snap/tag

    * extract switch lane logic from checkout-version.ts to switch-lanes.ts

    * consolidate snap-model-component into tag-model-component

    * clear confusion between remoteHead and laneHeadRemote

    * extract merge-lanes code to a new file merge-lanes.ts, refactor RemoteLane to use RemoteLaneId whenever possible

    * extract the diverge-data into a new file, add a new prop "remoteHead" and avoid adding the master remote into laneRemoteHead

    * extract the logic of version traversal into a new file

    * remove the hack of reseting the model-component heads

    * fix some forward compatibility issues with index.json

    * fix RemoteLaneId to indicate that a scope is mandatory

    * change ModelComponent.snaps.head to ModelComponent.head

    * fix bug when merging components to master

    * change variables with snapHead to head

commit bd1a2a1
Author: Ran Mizrahi <ranm8@users.noreply.github.com>
Date:   Fri Jul 17 02:45:50 2020 +0300

    component history working (#2851)

    Co-authored-by: Ran Mizrahi <ran@bit.dev>

commit b061300
Author: Ran Mizrahi <ranm8@users.noreply.github.com>
Date:   Thu Jul 16 14:30:40 2020 +0300

    Harmony/component load (#2840)

    - change component model
    - load components from the scope
    - update scope component with state from workspace
    - transform many workspace methods to use new ComponentID as arg
    - do not mutate original components from "onTag" envs' services

    Co-authored-by: Ran Mizrahi <ran@bit.dev>
    Co-authored-by: Gilad Shoham <shoham.gilad@gmail.com>

commit 1e6a8ac
Author: Gilad Shoham <shoham.gilad@gmail.com>
Date:   Wed Jul 15 23:45:40 2020 +0300

    fix windows e2e tests (#2846)

commit 73fb3c3
Author: Gilad Shoham <shoham.gilad@gmail.com>
Date:   Wed Jul 15 22:19:02 2020 +0300

    Harmony/workspace ui (#2835)

    First version of workspace ui

    Co-authored-by: Gilad Shoham <shoham.gilad@gmail.com>
    Co-authored-by: Ran Mizrahi <ran@bit.dev>
    Co-authored-by: Uri Kutner <urikutner@gmail.com>
    Co-authored-by: oded <oded@bit.dev>
    Co-authored-by: David First <davidnet@gmail.com>

commit bc90436
Author: David First <davidnet@gmail.com>
Date:   Mon Jul 13 23:08:05 2020 -0400

    improve bit link --rewire to preserve the internal paths (#2839)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants