diff --git a/README.md b/README.md index ec1905d..c9e2c4d 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Subversion Offline Solution (SOS 1.3.4) # +# Subversion Offline Solution (SOS 1.3.6) # [![Travis badge](https://travis-ci.org/ArneBachmann/sos.svg?branch=master)](https://travis-ci.org/ArneBachmann/sos) [![Build status](https://ci.appveyor.com/api/projects/status/fe915rtx02buqe4r?svg=true)](https://ci.appveyor.com/project/ArneBachmann/sos) @@ -9,6 +9,9 @@ - [Documentation](http://sos-vcs.net) (official website), [Code Repository](https://github.com/ArneBachmann/sos) (at Github) - [Buy a coffee](http://PayPal.Me/ArneBachmann/) for the developer to show your appreciation! +### Important notice from the author ### +> I have been developing this software over the course of the last 4 months in my spare time, and I put probably around 200 concentrated and sometimes painful hours into it, a rough equivalent of at 10.000€ of development costs that I have granted to the open source community. This project has taken a lot of time away from my family and important duties in my life. I have decided that I cannot continue at the current pace, unless getting support in form of a lifely SOS community, or by getting funding for for the time i put into SOS. I will continue to contribute bug fixes and add features according to my own priorities, unless someone else comes with feature requests and means to support them. I hope you understand and support SOS, which is really meant as a personal productivity tool. + ### List of Abbreviations and Definitions ### - **MPL**: [*Mozilla Public License*](https://www.mozilla.org/en-US/MPL/) - **PyPI**: [*Python Package Index*](https://pypi.python.org/pypi) @@ -58,6 +61,11 @@ SOS supports three different file handling models that you may use to your likin ## Latest Changes ## +- Version 1.4 release on 2018-02-xx: + - [Feature]() Introduces automatic upgrade for metadata format, making manual migration of previous and future releases obsolete + - [Feature]() Introduces experimental code for very fast branching. Use `sos branch [ []] --last --fast` for instant branching using only a reference to the parent on the new branch. This feature goes a step into the direction of Git and introduces complexity into the code base, but was seen as essential to not stand in the way of the developer. The burden of copying revisions to dependant branches is delayed to the point in time when the parent branch is destroyed, assuming that destroying a branch is much less often used than branching + - [Enhancement]() Reduced lines of code by relying on latest enhancements in Coconut (e.g. `typing` imports), plus removing obsolete code + - [Enhancement]() Allow to make the behavior of the `status` command configurable via `useChangesCommand=yes` to either show file tree status (the new default, mimicking SVN and Git) or the repository and branches status (sticking to use `changes` for file tree status instead, for people coming from Fossil) - Version 1.3 release on 2018-02-10: - [Bug 167](https://github.com/ArneBachmann/sos/issues/167) Accidentally crawling file tree and all revisions on status - [Enhancement 152, 162](https://github.com/ArneBachmann/sos/issues/152) PEP528/529 compatibility: Now working with any console encoding and file system encoding on Windows (at least with Python 3.6+) @@ -158,10 +166,10 @@ By means of the `sos config set ` command, you can set these flags ### Available Configuration Settings ### - `strict`: Flag for always performing full file comparsion, not relying on modification timestamp only; file size is always checked in both modes. Default: False - `track`: Flag for always going offline in tracking mode (SVN-style). Default: False -- `picky`: Flag for always going offline in picky mode (Git-styly). Default: False +- `picky`: Flag for always going offline in picky mode (Git-style). Default: False - `compress`: Flag for compressing versioned artifacts. Default: False - `defaultbranch`: Name of the initial branch created when going offline. Default: Dynamic per type of VCS in current working directory (e.g. `master` for Git, `trunk` for SVN, no name for Fossil) -- `texttype`: List of file patterns that should be recognized as text files that can be merged through textual diff, in addition to what Python's `mimetypes` library will detect as a `text/...` mime. *Default*: Empty list +- `texttype`: List of file patterns that should be recognized as text files that can be merged through textual diff, in addition to what Python's `mimetypes` library will detect as a `text/...` mime. Example: `*.bak` could be a text file on your system, so add it to the `texttype` configuration, either globally (default) or locally (using `--local`). *Default*: Empty list - `bintype`: List of file patterns that should be recognized as binary files which cannot be merged textually, overriding potential matches in `texttype`. Default: Empty list - `ignores`: List of filename patterns (without folder path) to ignore during repository operations. Any match from the corresponding white list will negate any hit for `ignores`. Default: See source code, e.g. `["*.bak", "*.py[cdo]]"` - `ignoresWhitelist`: List of filename patterns to be consider even if matched by an entry in the `ignores` list. Default: Empty list @@ -175,6 +183,11 @@ By means of the `sos config set ` command, you can set these flags - `sos update` will **not warn** if local changes are present! This is a noteworthy exception to the failsafe approach taken for most other commands +## Recipes ## +- Diff between any two revisions: Switch to the revision you want to compare against, then perform a diff with the other revision as argument +- Ignore whitespaces during diff: Add the option `--nows` or `--ignore-whitespace` + + ## Hints and Tipps ## - To migrate an offline repository, either use the `sos dump .sos.zip` command, or simple move the `.sos` folder into an (empty) target folder, and run `sos switch trunk --force` (or use whatever branch name you're wanting to recreate). For compressed offline repositories, you may simply `tar` all files, otherwise you may want to create an compressed archive for transferring the `.sos` folder - To save space when going offline, use the option `sos offline --compress`: It may increase commit times by a larger factor (e.g. 10x), but will also reduce the amount of storage needed to version files. To enable this option for all offline repositories, use `sos config set compress on` diff --git a/setup.py b/setup.py index 9ada96d..e5d2775 100644 --- a/setup.py +++ b/setup.py @@ -4,7 +4,7 @@ import os, shutil, subprocess, sys, time, unittest from setuptools import setup, find_packages -RELEASE = "1.3.4" +RELEASE = "1.3.6" print("sys.argv is %r" % sys.argv) readmeFile = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'README.md') @@ -12,7 +12,7 @@ print("Transpiling Coconut files to Python...") cmd = "-develop" if 0 == subprocess.Popen("coconut-develop --help", shell = True, stdout = subprocess.PIPE, stderr = subprocess.PIPE, bufsize = 10000000).wait() and os.getenv("NODEV", "false").strip().lower() != "true" else "" - assert 0 == os.system("coconut%s %s %s -l -t 3.4 sos %s" % (cmd, "-p" if not "--mypy" in sys.argv else "", "--force" if "--force" in sys.argv else "", "--mypy --ignore-missing-imports --warn-incomplete-stub --warn-redundant-casts --warn-return-any --warn-unused-ignores" if "--mypy" in sys.argv else "")) + assert 0 == os.system("coconut%s %s %s -l -t 3.4 sos %s" % (cmd, "-p" if not "--mypy" in sys.argv else "", "--force" if "--force" in sys.argv else "", "--mypy --ignore-missing-imports --warn-incomplete-stub --warn-redundant-casts --warn-unused-ignores" if "--mypy" in sys.argv else "")) # or useChanges try: sys.argv.remove('--mypy') except: pass try: sys.argv.remove('--force') diff --git a/sos/sos.coco b/sos/sos.coco index 5bcca25..48d401c 100644 --- a/sos/sos.coco +++ b/sos/sos.coco @@ -3,11 +3,8 @@ # Standard modules import codecs, collections, fnmatch, json, logging, mimetypes, os, shutil, sys, time -try: - from typing import Any, Dict, FrozenSet, IO, Iterator, List, Set, Tuple, Type, Union # only required for mypy -except: pass # typing not available (prior Python 3.5) sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -try: +try: # try needed as paths differ when installed via pip TODO investigate further import sos.version as version from sos.utility import * from sos.usage import * @@ -17,12 +14,10 @@ except: from usage import * # External dependencies -import configr # optional dependency -#from enforce import runtime_validation # https://github.com/RussBaz/enforce TODO doesn't work for "data" types +import configr # Constants -defaults = Accessor({"strict": False, "track": False, "picky": False, "compress": False, "texttype": ["*.md"], "bintype": [], "ignoreDirs": [".*", "__pycache__"], "ignoreDirsWhitelist": [], "ignores": ["__coconut__.py", "*.bak", "*.py[cdo]", "*.class", ".fslckout", "_FOSSIL_", "*.sos.zip"], "ignoresWhitelist": []}) termWidth = getTermWidth() - 1 # uses curses or returns conservative default of 80 APPNAME:str = "Subversion Offline Solution V%s (C) Arne Bachmann" % version.__release_version__ @@ -30,7 +25,7 @@ APPNAME:str = "Subversion Offline Solution V%s (C) Arne Bachmann" % version.__re # Functions def loadConfig() -> configr.Configr = # Accessor when using defaults only ''' Simplifies loading user-global config from file system or returning application defaults. ''' - config:configr.Configr = configr.Configr("sos", defaults = defaults) # defaults are used if key is not configured, but won't be saved + config:configr.Configr = configr.Configr(COMMAND, defaults = defaults) # defaults are used if key is not configured, but won't be saved f, g = config.loadSettings(clientCodeLocation = os.path.abspath(__file__), location = os.environ.get("TEST", None)) # latter for testing only if f is None: debug("Encountered a problem while loading the user configuration: %r" % g) config @@ -40,7 +35,6 @@ def saveConfig(config:configr.Configr) -> Tuple[str?, Exception?] = # Main data class -#@runtime_validation class Metadata: ''' This class doesn't represent the entire repository state in memory, but serves as a container for different repo operations, @@ -54,7 +48,7 @@ class Metadata: _.branch:int? = None # current branch number _.branches:Dict[int,BranchInfo] = {} # branch number zero represents the initial state at branching _.repoConf:Dict[str,Any] = {} - _.track:bool; _.picky:bool; _.strict:bool; _.compress:bool + _.track:bool; _.picky:bool; _.strict:bool; _.compress:bool; _.version:str?; _.format:int? _.loadBranches(offline = offline) # loads above values from repository, or uses application defaults _.commits:Dict[int,CommitInfo] = {} # consecutive numbers per branch, starting at 0 @@ -75,38 +69,44 @@ class Metadata: if len(changes.modifications) > 0: printo(ajoin("MOD ", sorted(changes.modifications.keys()), "\n")) def loadBranches(_, offline:bool = False): - ''' Load list of branches and current branch info from metadata file. ''' + ''' Load list of branches and current branch info from metadata file. offline = offline command avoids message. ''' try: # fails if not yet created (on initial branch/commit) - branches:List[Tuple] + branches:List[List] # deserialized JSON is only list, while the real type of _.branches is a dict number -> BranchInfo (Coconut data type/named tuple) with codecs.open(encode(os.path.join(_.root, metaFolder, metaFile)), "r", encoding = UTF8) as fd: repo, branches, config = json.load(fd) _.tags = repo["tags"] # list of commit messages to treat as globally unique tags _.branch = repo["branch"] # current branch integer - _.track, _.picky, _.strict, _.compress, _.version = [repo[r] for r in ["track", "picky", "strict", "compress", "version"]] + _.track, _.picky, _.strict, _.compress, _.version, _.format = [repo.get(r, None) for r in ["track", "picky", "strict", "compress", "version", "format"]] upgraded:List[str] = [] - if repo["version"] < "2018.1210.3028": # For older versions, see https://pypi.python.org/simple/sos-vcs/ - branches[:] = [branch + ([[]] if len(branch) < 6 else []) for branch in branches] # add untracking information + if _.version is None: + _.version = "0 - pre-1.2" + upgraded.append("pre-1.2") + if len(branches[0]) < 6: # For older versions, see https://pypi.python.org/simple/sos-vcs/ + branches[:] = [branch + [[]] * (6 - len(branch)) for branch in branches] # add untracking information, if missing upgraded.append("2018.1210.3028") - if "format" not in repo: # must be pre-1.4 - repo["format"] = 1 # 1.4+ - branches[:] = [branch + [None] for branch in branches] # adds empty branching point information - upgraded.append("1.4") + if _.format is None: # must be before 1.3.5+ + _.format = METADATA_FORMAT # marker for first metadata file format + branches[:] = [branch + [None] * (8 - len(branch)) for branch in branches] # adds empty branching point information (branch/revision) + upgraded.append("1.3.5") _.branches = {i.number: i for i in (BranchInfo(*item) for item in branches)} # re-create type info _.repoConf = config if upgraded: for upgrade in upgraded: warn("!!! Upgraded repository metadata to match SOS version %r" % upgrade) - warn("To undo upgrade%s, restore metadata *now* from '%s/%s'" % ("s" if len(upgraded) > 1 else "", metaFolder, metaBack)) + warn("To revert the metadata upgrade%s, restore %s/%s from %s/%s NOW" % ("s" if len(upgraded) > 1 else "", metaFolder, metaFile, metaFolder, metaBack)) _.saveBranches() - except Exception as E: # if not found, create metadata folder + except Exception as E: # if not found, create metadata folder with default values _.branches = {} - _.track, _.picky, _.strict, _.compress, _.version = [defaults[k] for k in ["track", "picky", "strict", "compress"]] + [version.__version__] + _.track, _.picky, _.strict, _.compress, _.version, _.format = [defaults[k] for k in ["track", "picky", "strict", "compress"]] + [version.__version__, METADATA_FORMAT] (debug if offline else warn)("Couldn't read branches metadata: %r" % E) def saveBranches(_, also:Dict[str,Any] = {}): ''' Save list of branches and current branch info to metadata file. ''' - tryOrIgnore(() -> shutil.copy2(encode(os.path.join(_.root, metaFolder, metaFile)), encode(os.path.join(_.root, metaFolder, metaBack)))) + tryOrIgnore(() -> shutil.copy2(encode(os.path.join(_.root, metaFolder, metaFile)), encode(os.path.join(_.root, metaFolder, metaBack)))) # backup with codecs.open(encode(os.path.join(_.root, metaFolder, metaFile)), "w", encoding = UTF8) as fd: - store:Dict[str,Any] = {"format": 1, "version": _.version, "tags": _.tags, "branch": _.branch, "track": _.track, "picky": _.picky, "strict": _.strict, "compress": _.compress} + store:Dict[str,Any] = { + "tags": _.tags, "branch": _.branch, + "track": _.track, "picky": _.picky, "strict": _.strict, "compress": _.compress, "version": _.version, "format": METADATA_FORMAT # HINT uses _.version instead of constant to allow the upgrade procedure to write a specific version + } store.update(also) # allows overriding certain values at certain points in time json.dump((store, list(_.branches.values()), _.repoConf), fd, ensure_ascii = False) # stores using unicode codepoints, fd knows how to encode them @@ -128,43 +128,53 @@ class Metadata: def loadBranch(_, branch:int): ''' Load all commit information from a branch meta data file. ''' - with codecs.open(encode(os.path.join(_.root, metaFolder, "b%d" % branch, metaFile)), "r", encoding = UTF8) as fd: + with codecs.open(encode(branchFolder(branch, file = metaFile)), "r", encoding = UTF8) as fd: commits:List[List[Any]] = json.load(fd) # list of CommitInfo that needs to be unmarshalled into value types _.commits = {i.number: i for i in (CommitInfo(*item) for item in commits)} # re-create type info _.branch = branch def saveBranch(_, branch:int): - ''' Save all commit information to a branch meta data file. ''' - tryOrIgnore(() -> shutil.copy2(encode(os.path.join(_.root, metaFolder, "b%d" % branch, metaFile)), encode(os.path.join(_.root, metaFolder, metaBack)))) - with codecs.open(encode(os.path.join(_.root, metaFolder, "b%d" % branch, metaFile)), "w", encoding = UTF8) as fd: + ''' Save all commits to a branch meta data file. ''' + tryOrIgnore(() -> shutil.copy2(encode(branchFolder(branch, file = metaFile)), encode(branchFolder(branch, metaBack)))) # backup + with codecs.open(encode(branchFolder(branch, file = metaFile)), "w", encoding = UTF8) as fd: json.dump(list(_.commits.values()), fd, ensure_ascii = False) - def duplicateBranch(_, branch:int, name:str?): - ''' Create branch from an existing branch/revision. WARN: Caller must have loaded branches information. - branch: target branch (should not exist yet) - name: optional name of new branch + def duplicateBranch(_, branch:int, name:str? = None, initialMessage:str? = None, full:bool = True): + ''' Create branch from an existing branch/revision. + In case of full branching, copy all revisions, otherwise create only reference to originating branch/revision. + branch: new target branch number (must not exist yet) + name: optional name of new branch (currently always set by caller) + initialMessage: message for commit if not last and file tree modified + full: always create full branch copy, don't use a parent reference _.branch: current branch ''' debug("Duplicating branch '%s' to '%s'..." % (_.branches[_.branch].name ?? ("b%d" % _.branch), (name ?? "b%d" % branch))) - tracked:List[str] = [t for t in _.branches[_.branch].tracked] # copy - untracked:List[str] = [u for u in _.branches[_.branch].untracked] - os.makedirs(encode(branchFolder(branch, 0, base = _.root))) - _.loadBranch(_.branch) + now:int = int(time.time() * 1000) + _.loadBranch(_.branch) # load commits for current (originating) branch revision:int = max(_.commits) - _.computeSequentialPathSet(_.branch, revision) # full set of files in revision to _.paths - for path, pinfo in _.paths.items(): - _.copyVersionedFile(_.branch, revision, branch, 0, pinfo) - _.commits = {0: CommitInfo(0, int(time.time() * 1000), "Branched from '%s'" % (_.branches[_.branch].name ?? "b%d" % _.branch))} # store initial commit - _.saveBranch(branch) # save branch meta data to branch folder - _.saveCommit(branch, 0) # save commit meta data to revision folder - _.branches[branch] = BranchInfo(branch, _.commits[0].ctime, name, _.branches[_.branch].inSync, tracked, untracked) # save branch info, before storing repo state at caller + _.commits.clear() + newBranch:BranchInfo = dataCopy(BranchInfo, _.branches[_.branch], + number = branch, ctime = now, name = name ?? "Branched from '%s'" % (_.branches[_.branch].name ?? "b%d" % _.branch), + tracked = [t for t in _.branches[_.branch].tracked], untracked = [u for u in _.branches[_.branch].untracked], + parent = None if full else _.branch, revision = None if full else revision + ) + os.makedirs(encode(revisionFolder(branch, 0, base = _.root) if full else branchFolder(branch, base = _.root))) + if full: # not fast branching via reference - copy all current files to new branch + _.computeSequentialPathSet(_.branch, revision) # full set of files in latest revision in _.paths + for path, pinfo in _.paths.items(): _.copyVersionedFile(_.branch, revision, branch, 0, pinfo) # copy into initial branch revision + _.commits[0] = CommitInfo(0, now, initialMessage ?? "Branched from '%s'" % (_.branches[_.branch].name ?? "b%d" % _.branch)) # store initial commit TODO also contain message from latest revision of originating branch + _.saveCommit(branch, 0) # save commit meta data to revision folder + _.saveBranch(branch) # save branch meta data to branch folder - for fast branching, only empty dict + _.branches[branch] = newBranch # save branches meta data, needs to be saved in caller code def createBranch(_, branch:int, name:str? = None, initialMessage:str? = None): - ''' Create a new branch from current file tree. This clears all known commits and modifies the file system. - branch: target branch (should not exist yet) + ''' Create a new branch from the current file tree. This clears all known commits and modifies the file system. + branch: target branch number (must not exist yet) name: optional name of new branch + initialMessage: commit message for revision 0 of the new branch _.branch: current branch, must exist already ''' + now:int = int(time.time() * 1000) simpleMode = not (_.track or _.picky) tracked:List[str] = [t for t in _.branches[_.branch].tracked] if _.track and len(_.branches) > 0 else [] # in case of initial branch creation untracked:List[str] = [t for t in _.branches[_.branch].untracked] if _.track and len(_.branches) > 0 else [] @@ -176,44 +186,59 @@ class Metadata: if msg: printo(msg) # display compression factor _.paths.update(changes.additions.items()) else: # tracking or picky mode: branch from latest revision - os.makedirs(encode(branchFolder(branch, 0, base = _.root))) + os.makedirs(encode(revisionFolder(branch, 0, base = _.root))) if _.branch is not None: # not immediately after "offline" - copy files from current branch _.loadBranch(_.branch) revision:int = max(_.commits) # TODO what if last switch was to an earlier revision? no persisting of last checkout _.computeSequentialPathSet(_.branch, revision) # full set of files in revision to _.paths for path, pinfo in _.paths.items(): _.copyVersionedFile(_.branch, revision, branch, 0, pinfo) - ts = int(time.time() * 1000) - _.commits = {0: CommitInfo(0, ts, initialMessage ?? "Branched on %s" % strftime(ts))} # store initial commit for new branch + _.commits = {0: CommitInfo(0, now, initialMessage ?? "Branched on %s" % strftime(now))} # store initial commit for new branch _.saveBranch(branch) # save branch meta data (revisions) to branch folder _.saveCommit(branch, 0) # save commit meta data to revision folder _.branches[branch] = BranchInfo(branch, _.commits[0].ctime, name, True if len(_.branches) == 0 else _.branches[_.branch].inSync, tracked, untracked) # save branch info, in case it is needed def removeBranch(_, branch:int) -> BranchInfo = ''' Entirely remove a branch and all its revisions from the file system. ''' - shutil.rmtree(encode(os.path.join(_.root, metaFolder, "b%d" % branch))) # TODO put into recycle bin, under ./sos? - binfo = _.branches[branch] + binfo:BranchInfo + deps:List[Tuple[int,int]] = [(binfo.number, binfo.revision) for binfo in _.branches.values() if binfo.parent is not None and _.getParentBranch(binfo.number, 0) == branch] # get transitively depending branches + if deps: # need to copy all parent revisions to dependet branches first + minrev:int = min([e[1] for e in deps]) # minimum revision ever branched from parent (ignoring transitive branching!) + progress:indicator = ProgressIndicator() + for rev in range(0, minrev + 1): # rely on caching by copying revision-wise as long as needed in all depending branches + for dep, _rev in deps: + if rev <= _rev: + printo("\rIntegrating revision %02d into dependant branch %02d %s" % (rev, dep, progress.getIndicator())) + shutil.copytree(encode(revisionFolder(branch, rev, base = _.root)), encode(revisionFolder(dep, rev, base = _.root))) # folder would not exist yet + for dep, _rev in deps: # copy remaining revisions per branch + for rev in range(minrev + 1, _rev + 1): + printo("\rIntegrating revision %02d into dependant branch %02d %s" % (rev, dep, progress.getIndicator())) + shutil.copytree(encode(revisionFolder(_.getParentBranch(dep, rev), rev, base = _.root)), encode(revisionFolder(dep, rev, base = _.root))) + _.branches[dep] = dataCopy(BranchInfo, _.branches[dep], parent = None, revision = None) # remove reference information + printo(" " * termWidth + "\r") + tryOrIgnore(() -> shutil.rmtree(encode(branchFolder(branch) + BACKUP_SUFFIX))) # remove previous backup first + os.rename(encode(branchFolder(branch)), encode(branchFolder(branch) + BACKUP_SUFFIX)) + binfo = _.branches[branch] # keep reference for caller del _.branches[branch] - _.branch = max(_.branches) + _.branch = max(_.branches) # switch to another valid branch _.saveBranches() _.commits.clear() binfo def loadCommit(_, branch:int, revision:int): - ''' Load all file information from a commit meta data. ''' - with codecs.open(encode(branchFolder(branch, revision, base = _.root, file = metaFile)), "r", encoding = UTF8) as fd: - _.paths = json.load(fd) + ''' Load all file information from a commit meta data; if branched from another branch before specified revision, load correct revision recursively. ''' + _branch:int = _.getParentBranch(branch, revision) + with codecs.open(encode(revisionFolder(_branch, revision, base = _.root, file = metaFile)), "r", encoding = UTF8) as fd: _.paths = json.load(fd) _.paths = {path: PathInfo(*item) for path, item in _.paths.items()} # re-create type info - _.branch = branch + _.branch = branch # store current branch information = "switch" to loaded branch/commit def saveCommit(_, branch:int, revision:int): ''' Save all file information to a commit meta data file. ''' - target:str = branchFolder(branch, revision, base = _.root) + target:str = revisionFolder(branch, revision, base = _.root) try: os.makedirs(encode(target)) except: pass - tryOrIgnore(() -> shutil.copy2(encode(os.path.join(target, metaFile)), encode(os.path.join(target, metaBack)))) - with codecs.open(encode(os.path.join(target, metaFile)), "w", encoding = UTF8) as fd: - json.dump(_.paths, fd, ensure_ascii = False) + tryOrIgnore(() -> shutil.copy2(encode(os.path.join(target, metaFile)), encode(os.path.join(target, metaBack)))) # ignore error for first backup + with codecs.open(encode(os.path.join(target, metaFile)), "w", encoding = UTF8) as fd: json.dump(_.paths, fd, ensure_ascii = False) def findChanges(_, branch:int? = None, revision:int? = None, checkContent:bool = False, inverse:bool = False, considerOnly:FrozenSet[str]? = None, dontConsider:FrozenSet[str]? = None, progress:bool = False) -> Tuple[ChangeSet,str?] = ''' Find changes on the file system vs. in-memory paths (which should reflect the latest commit state). @@ -227,7 +252,7 @@ class Metadata: ''' write = branch is not None and revision is not None if write: - try: os.makedirs(encode(branchFolder(branch, revision, base = _.root))) + try: os.makedirs(encode(revisionFolder(branch, revision, base = _.root))) except FileExistsError: pass # HINT "try" only necessary for *testing* hash collision code (!) TODO probably raise exception otherwise in any case? changes:ChangeSet = ChangeSet({}, {}, {}, {}) # TODO Needs explicity initialization due to mypy problems with default arguments :-( indicator:ProgressIndicator? = ProgressIndicator() if progress else None # optional file list progress indicator @@ -262,7 +287,7 @@ class Metadata: if filename not in _.paths: # detected file not present (or untracked) in (other) branch nameHash = hashStr(filename) try: - hashed, written = hashFile(filepath, _.compress, saveTo = branchFolder(branch, revision, base = _.root, file = nameHash) if write else None, callback = None if not progress else (sign) -> printo(outstring + " " + sign + " " * max(0, termWidth - len(outstring) - 2), nl = "")) if size > 0 else (None, 0) + hashed, written = hashFile(filepath, _.compress, saveTo = revisionFolder(branch, revision, base = _.root, file = nameHash) if write else None, callback = ((sign) -> printo(outstring + " " + sign + " " * max(0, termWidth - len(outstring) - 2), nl = "")) if show else None) if size > 0 else (None, 0) changes.additions[filename] = PathInfo(nameHash, size, mtime, hashed) compressed += written; original += size except Exception as E: exception(E) @@ -270,12 +295,12 @@ class Metadata: last = _.paths[filename] # filename is known - check for modifications if last.size is None: # was removed before but is now added back - does not apply for tracking mode (which never marks files for removal in the history) try: - hashed, written = hashFile(filepath, _.compress, saveTo = branchFolder(branch, revision, base = _.root, file = last.nameHash) if write else None, callback = None if not progress else (sign) -> printo(outstring + " " + sign + " " * max(0, termWidth - len(outstring) - 2), nl = "")) if size > 0 else (None, 0) + hashed, written = hashFile(filepath, _.compress, saveTo = revisionFolder(branch, revision, base = _.root, file = last.nameHash) if write else None, callback = None if not progress else (sign) -> printo(outstring + " " + sign + " " * max(0, termWidth - len(outstring) - 2), nl = "")) if size > 0 else (None, 0) changes.additions[filename] = PathInfo(last.nameHash, size, mtime, hashed); continue except Exception as E: exception(E) elif size != last.size or (not checkContent and mtime != last.mtime) or (checkContent and tryOrDefault(() -> (hashFile(filepath, _.compress)[0] != last.hash), default = False)): # detected a modification TODO wrap hashFile exception try: - hashed, written = hashFile(filepath, _.compress, saveTo = branchFolder(branch, revision, base = _.root, file = last.nameHash) if write else None, callback = None if not progress else (sign) -> printo(outstring + " " + sign + " " * max(0, termWidth - len(outstring) - 2), nl = "")) if (last.size if inverse else size) > 0 else (last.hash if inverse else None, 0) + hashed, written = hashFile(filepath, _.compress, saveTo = revisionFolder(branch, revision, base = _.root, file = last.nameHash) if write else None, callback = None if not progress else (sign) -> printo(outstring + " " + sign + " " * max(0, termWidth - len(outstring) - 2), nl = "")) if (last.size if inverse else size) > 0 else (last.hash if inverse else None, 0) changes.modifications[filename] = PathInfo(last.nameHash, last.size if inverse else size, last.mtime if inverse else mtime, hashed) except Exception as E: exception(E) else: continue # with next file @@ -305,13 +330,17 @@ class Metadata: if incrementally: yield _.paths m:Metadata = Metadata(_.root); rev:int # next changes TODO avoid loading all metadata and config for rev in range(1, revision + 1): - m.loadCommit(branch, rev) + m.loadCommit(_.getParentBranch(branch, rev), rev) for p, info in m.paths.items(): if info.size == None: del _.paths[p] else: _.paths[p] = info if incrementally: yield _.paths yield None # for the default case - not incrementally + def getTrackingPatterns(_, branch:int? = None, negative:bool = False) -> FrozenSet[str] = + ''' Returns list of tracking patterns (or untracking patterns if negative) for provided branch or current branch. ''' + f{} if not (_.track or _.picky) else frozenset(_.branches[branch ?? _.branch].untracked if negative else _.branches[branch ?? _.branch].tracked) + def parseRevisionString(_, argument:str) -> Tuple[int?,int?]: ''' Commit identifiers can be str or int for branch, and int for revision. Revision identifiers can be negative, with -1 being last commit. @@ -334,21 +363,29 @@ class Metadata: def findRevision(_, branch:int, revision:int, nameHash:str) -> Tuple[int,str] = while True: # find latest revision that contained the file physically - source:str = branchFolder(branch, revision, base = _.root, file = nameHash) + _branch:int = _.getParentBranch(branch, revision) + source:str = revisionFolder(_branch, revision, base = _.root, file = nameHash) if os.path.exists(encode(source)) and os.path.isfile(source): break revision -= 1 if revision < 0: Exit("Cannot determine versioned file '%s' from specified branch '%d'" % (nameHash, branch)) revision, source + def getParentBranch(_, branch:int, revision:int) -> int = + ''' Determine originating branch for a (potentially branched) revision, traversing all branch parents until found. ''' + other:int? = _.branches[branch].parent # reference to originating parent branch, or None + if other is None or revision > _.branches[branch].revision: return branch # need to load commit from other branch instead + while _.branches[other].parent is not None and revision <= _.branches[other].revision: other = _.branches[other].parent + other + def copyVersionedFile(_, branch:int, revision:int, toBranch:int, toRevision:int, pinfo:PathInfo): ''' Copy versioned file to other branch/revision. ''' - target:str = branchFolder(toBranch, toRevision, base = _.root, file = pinfo.nameHash) + target:str = revisionFolder(toBranch, toRevision, base = _.root, file = pinfo.nameHash) revision, source = _.findRevision(branch, revision, pinfo.nameHash) shutil.copy2(encode(source), encode(target)) def readOrCopyVersionedFile(_, branch:int, revision:int, nameHash:str, toFile:str? = None) -> bytes? = ''' Return file contents, or copy contents into file path provided. ''' - source:str = branchFolder(branch, revision, base = _.root, file = nameHash) + source:str = revisionFolder(_.getParentBranch(branch, revision), revision, base = _.root, file = nameHash) try: with openIt(source, "r", _.compress) as fd: if toFile is None: return fd.read() # read bytes into memory and return @@ -384,13 +421,9 @@ class Metadata: except Exception as E: error("Cannot update file's timestamp after restoration '%r'" % E) None - def getTrackingPatterns(_, branch:int? = None, negative:bool = False) -> FrozenSet[str] = - ''' Returns list of tracking patterns (or untracking patterns if negative) for provided branch or current branch. ''' - f{} if not (_.track or _.picky) else frozenset(_.branches[branch ?? _.branch].untracked if negative else _.branches[branch ?? _.branch].tracked) - # Main client operations -def offline(argument:str? = None, options:str[] = []): +def offline(name:str? = None, initialMessage:str? = None, options:str[] = []): ''' Initial command to start working offline. ''' if os.path.exists(encode(metaFolder)): if '--force' not in options: Exit("Repository folder is either already offline or older branches and commits were left over.\nUse 'sos online' to check for dirty branches, or\nWipe existing offline branches with 'sos offline --force'") @@ -406,7 +439,7 @@ def offline(argument:str? = None, options:str[] = []): elif '--track' in options or m.c.track: m.track = True # Svn-like if '--strict' in options or m.c.strict: m.strict = True # always hash contents debug(MARKER + "Going offline...") - m.createBranch(0, argument ?? defaults["defaultbranch"], initialMessage = "Offline repository created on %s" % strftime()) # main branch's name may be None (e.g. for fossil) + m.createBranch(0, name ?? defaults["defaultbranch"], initialMessage ?? "Offline repository created on %s" % strftime()) # main branch's name may be None (e.g. for fossil) m.branch = 0 m.saveBranches(also = {"version": version.__version__}) # stores version info only once. no change immediately after going offline, going back online won't issue a warning info(MARKER + "Offline repository prepared. Use 'sos online' to finish offline work") @@ -416,10 +449,13 @@ def online(options:str[] = []): debug(MARKER + "Going back online...") force:bool = '--force' in options m:Metadata = Metadata() + strict:bool = '--strict' in options or m.strict m.loadBranches() if any([not b.inSync for b in m.branches.values()]) and not force: Exit("There are still unsynchronized (dirty) branches.\nUse 'sos log' to list them.\nUse 'sos commit' and 'sos switch' to commit dirty branches to your VCS before leaving offline mode.\nUse 'sos online --force' to erase all aggregated offline revisions") - strict:bool = '--strict' in options or m.strict + m.loadBranch(m.branch) + maxi:int = max(m.commits) if m.commits else m.branches[m.branch].revision # one commit guaranteed for first offline branch, for fast-branched branches a revision in branchinfo if options.count("--force") < 2: + m.computeSequentialPathSet(m.branch, maxi) # load all commits up to specified revision changes, msg = m.findChanges( checkContent = strict, considerOnly = None if not (m.track or m.picky) else m.getTrackingPatterns(), @@ -430,44 +466,47 @@ def online(options:str[] = []): except Exception as E: Exit("Error removing offline repository: %r" % E) info(MARKER + "Offline repository removed, you're back online") -def branch(argument:str? = None, options:str[] = []): - ''' Create a new branch (from file tree or last revision) and (by default) continue working on it. ''' +def branch(name:str? = None, initialMessage:str? = None, options:str[] = []): + ''' Create a new branch (from file tree or last revision) and (by default) continue working on it. + Force not necessary, as either branching from last revision anyway, or branching file tree anyway. + ''' last:bool = '--last' in options # use last revision for branching, not current file tree - stay:bool = '--stay' in options # continue on current branch after branching - force:bool = '--force' in options # branch even with local modifications + stay:bool = '--stay' in options # continue on current branch after branching (don't switch) + fast:bool = '--fast' in options # branch by referencing TODO move to default and use --full instead for old behavior m:Metadata = Metadata() m.loadBranch(m.branch) - if argument and m.getBranchByName(argument) is not None: Exit("Branch '%s' already exists. Cannot proceed" % argument) # create a named branch + maxi:int = max(m.commits) if m.commits else m.branches[m.branch].revision + if name and m.getBranchByName(name) is not None: Exit("Branch '%s' already exists. Cannot proceed" % name) # attempted to create a named branch branch = max(m.branches.keys()) + 1 # next branch's key - this isn't atomic but we assume single-user non-concurrent use here - debug(MARKER + "Branching to %sbranch b%02d%s%s..." % ("unnamed " if argument is None else "", branch, " '%s'" % argument if argument else "", " from last revision" if last else "")) - if last: m.duplicateBranch(branch, (argument ?? "") + " (Branched from r%02d/b%02d)" % (m.branch, max(m.commits.keys()))) # branch from branch's last revision - else: m.createBranch(branch, argument ?? "Branched from r%02d/b%02d" % (m.branch, max(m.commits.keys()))) # branch from current file tree state - if not stay: - m.branch = branch - m.saveBranches() - info(MARKER + "%s new %sbranch b%02d%s" % ("Continue work after branching" if stay else "Switched to", "unnamed " if argument is None else "", branch, " '%s'" % argument if argument else "")) - -def changes(argument:str = None, options:str[] = [], onlys:FrozenSet[str]? = None, excps:FrozenSet[str]? = None) -> ChangeSet = + debug(MARKER + "Branching to %sbranch b%02d%s%s..." % ("unnamed " if name is None else "", branch, " '%s'" % name if name is not None else "", " from last revision" if last else "")) + if last: m.duplicateBranch(branch, name, (initialMessage + " " if initialMessage else "") + "(Branched from r%02d/b%02d)" % (m.branch, maxi), not fast) # branch from last revision + else: m.createBranch(branch, name, initialMessage ?? "Branched from file tree after r%02d/b%02d" % (m.branch, maxi)) # branch from current file tree state + if not stay: m.branch = branch + m.saveBranches() # TODO or indent again? + info(MARKER + "%s new %sbranch b%02d%s" % ("Continue work after branching" if stay else "Switched to", "unnamed " if name is None else "", branch, " '%s'" % name if name else "")) + +def changes(argument:str = None, options:str[] = [], onlys:FrozenSet[str]? = None, excps:FrozenSet[str]? = None, useChanges:bool = False) -> ChangeSet = ''' Show changes of file tree vs. (last or specified) revision on current or specified branch. ''' - if '--repo' in options: status(options, onlys, excps); return m:Metadata = Metadata(); branch:int?; revision:int? + if '--repo' in options or useChanges or m.c.useChangesCommand: status(options, onlys, excps); return # TODO for fossil not possible to restore SVN behavior strict:bool = '--strict' in options or m.strict branch, revision = m.parseRevisionString(argument) if branch not in m.branches: Exit("Unknown branch") m.loadBranch(branch) # knows commits - revision = revision if revision >= 0 else len(m.commits) + revision # negative indexing - if revision < 0 or revision > max(m.commits): Exit("Unknown revision r%02d" % revision) + revision = m.branches[branch].revision if not m.commits else (revision if revision >= 0 else max(m.commits) + 1 + revision) # negative indexing + if revision < 0 or (m.commits and revision > max(m.commits)): Exit("Unknown revision r%02d" % revision) debug(MARKER + "Changes of file tree vs. revision '%s/r%02d'" % (m.branches[branch].name ?? "b%02d" % branch, revision)) m.computeSequentialPathSet(branch, revision) # load all commits up to specified revision changes, msg = m.findChanges( checkContent = strict, - considerOnly = onlys if not (m.track or m.picky) else conditionalIntersection(onlys, m.getTrackingPatterns() | m.getTrackingPatterns(branch)), + considerOnly = onlys if not (m.track or m.picky) else conditionalIntersection(onlys, m.getTrackingPatterns() | m.getTrackingPatterns(branch)), dontConsider = excps if not (m.track or m.picky) else excps ?? (m.getTrackingPatterns(negative = True) | m.getTrackingPatterns(branch, negative = True)), progress = '--progress' in options) m.listChanges(changes) changes # for unit tests only TODO remove -def _diff(m:Metadata, branch:int, revision:int, changes:ChangeSet): # TODO introduce option to diff against committed revision +def _diff(m:Metadata, branch:int, revision:int, changes:ChangeSet, ignoreWhitespace:bool, textWrap:bool = False): # TODO introduce option to diff against committed revision + wrap:(str) -> str = ((s) -> s) if textWrap else ((s) -> s[:termWidth]) onlyBinaryModifications:ChangeSet = dataCopy(ChangeSet, changes, modifications = {k: v for k, v in changes.modifications.items() if not m.isTextType(os.path.basename(k))}) m.listChanges(onlyBinaryModifications) # only list modified binary files for path, pinfo in (c for c in changes.modifications.items() if m.isTextType(os.path.basename(c[0]))): # only consider modified text files @@ -476,43 +515,41 @@ def _diff(m:Metadata, branch:int, revision:int, changes:ChangeSet): # TODO intr else: content = m.restoreFile(None, branch, revision, pinfo); assert content is not None # versioned file abspath:str = os.path.normpath(os.path.join(m.root, path.replace(SLASH, os.sep))) # current file blocks:List[MergeBlock]; nl:bytes - blocks, nl = merge(filename = abspath, into = content, diffOnly = True) # only determine change blocks + blocks, nl = merge(filename = abspath, into = content, diffOnly = True, ignoreWhitespace = ignoreWhitespace) # only determine change blocks printo("DIF %s%s %s" % (path, " " if len(blocks) == 1 and blocks[0].tipe == MergeBlockType.KEEP else "", NL_NAMES[nl])) for block in blocks: # if block.tipe in [MergeBlockType.INSERT, MergeBlockType.REMOVE]: # pass # TODO print some previous and following lines - which aren't accessible here anymore if block.tipe == MergeBlockType.INSERT: # TODO show color via (n)curses or other library? - for no, line in enumerate(block.lines): - printo("--- %04d |%s|" % (no + block.line, line)) + for no, line in enumerate(block.lines): printo(wrap("--- %04d |%s|" % (no + block.line, line))) elif block.tipe == MergeBlockType.REMOVE: - for no, line in enumerate(block.lines): - printo("+++ %04d |%s|" % (no + block.line, line)) - elif block.tipe == MergeBlockType.REPLACE: # TODO for MODIFY also show intra-line change ranges (TODO remove if that code was also removed) - for no, line in enumerate(block.replaces.lines): - printo("- | %04d |%s|" % (no + block.replaces.line, line)) - for no, line in enumerate(block.lines): - printo("+ | %04d |%s|" % (no + block.line, line)) + for no, line in enumerate(block.lines): printo(wrap("+++ %04d |%s|" % (no + block.line, line))) + elif block.tipe == MergeBlockType.REPLACE: + for no, line in enumerate(block.replaces.lines): printo(wrap("- | %04d |%s|" % (no + block.replaces.line, line))) + for no, line in enumerate(block.lines): printo(wrap("+ | %04d |%s|" % (no + block.line, line))) # elif block.tipe == MergeBlockType.KEEP: pass # elif block.tipe == MergeBlockType.MOVE: # intra-line modifications + if block.tipe != MergeBlockType.KEEP: printo() def diff(argument:str = "", options:str[] = [], onlys:FrozenSet[str]? = None, excps:FrozenSet[str]? = None): ''' Show text file differences of file tree vs. (last or specified) revision on current or specified branch. ''' m:Metadata = Metadata(); branch:int?; revision:int? strict:bool = '--strict' in options or m.strict - _from:str? = {None: option.split("--from=")[1] for option in options if option.startswith("--from=")}.get(None, None) # TODO implement + ignoreWhitespace:bool = '--ignore-whitespace' in options or '--iw' in options + wrap:bool = '--wrap' in options # allow text to wrap around branch, revision = m.parseRevisionString(argument) # if nothing given, use last commit if branch not in m.branches: Exit("Unknown branch") m.loadBranch(branch) # knows commits - revision = revision if revision >= 0 else len(m.commits) + revision # negative indexing - if revision < 0 or revision > max(m.commits): Exit("Unknown revision r%02d" % revision) + revision = m.branches[branch].revision if not m.commits else (revision if revision >= 0 else max(m.commits) + 1 + revision) # negative indexing + if revision < 0 or (m.commits and revision > max(m.commits)): Exit("Unknown revision r%02d" % revision) debug(MARKER + "Textual differences of file tree vs. revision '%s/r%02d'" % (m.branches[branch].name ?? "b%02d" % branch, revision)) m.computeSequentialPathSet(branch, revision) # load all commits up to specified revision changes, msg = m.findChanges( checkContent = strict, inverse = True, considerOnly = onlys if not (m.track or m.picky) else conditionalIntersection(onlys, m.getTrackingPatterns() | m.getTrackingPatterns(branch)), - dontConsider = excps if not (m.track or m.picky) else excps ?? (m.getTrackingPatterns(negative = True) | m.getTrackingPatterns(branch, negative = True)), + dontConsider = excps if not (m.track or m.picky) else excps ?? (m.getTrackingPatterns(negative = True) | m.getTrackingPatterns(branch, negative = True)), progress = '--progress' in options) - _diff(m, branch, revision, changes) + _diff(m, branch, revision, changes, ignoreWhitespace = ignoreWhitespace, textWrap = wrap) def commit(argument:str? = None, options:str[] = [], onlys:FrozenSet[str]? = None, excps:FrozenSet[str]? = None): ''' Create new revision from file tree changes vs. last commit. ''' @@ -545,15 +582,16 @@ def status(options:str[] = [], onlys:FrozenSet[str]? = None, excps:FrozenSet[str info("Installation path: %s" % os.path.abspath(os.path.dirname(__file__))) info("Current SOS version: %s" % version.__version__) info("At creation version: %s" % m.version) + info("Metadata format: %s" % m.format) info("Content checking: %sactivated" % ("" if m.strict else "de")) info("Data compression: %sactivated" % ("" if m.compress else "de")) info("Repository mode: %s" % ("track" if m.track else ("picky" if m.picky else "simple"))) info("Number of branches: %d" % len(m.branches)) -# info("Revisions: %d" % sum([])) trackingPatterns:FrozenSet[str] = m.getTrackingPatterns() untrackingPatterns:FrozenSet[str] = m.getTrackingPatterns(negative = True) - m.loadBranch(m.branch) - m.computeSequentialPathSet(m.branch, max(m.commits)) # load all commits up to specified revision # line 508 + m.loadBranch(current) + maxi:int = max(m.commits) if m.commits else m.branches[m.branch].revision + m.computeSequentialPathSet(current, maxi) # load all commits up to specified revision # line 508 changes, msg = m.findChanges( checkContent = strict, considerOnly = onlys if not (m.track or m.picky) else conditionalIntersection(onlys, trackingPatterns), @@ -563,7 +601,8 @@ def status(options:str[] = [], onlys:FrozenSet[str]? = None, excps:FrozenSet[str sl:int = max([len(b.name ?? "") for b in m.branches.values()]) for branch in sorted(m.branches.values(), key = (b) -> b.number): m.loadBranch(branch.number) # knows commit history - printo(" %s b%02d%s @%s (%s) with %d commits%s" % ("*" if current == branch.number else " ", branch.number, ((" %%%ds" % (sl + 2)) % ("'%s'" % branch.name)) if branch.name else "", strftime(branch.ctime), "in sync" if branch.inSync else "dirty", len(m.commits), ". Last comment: '%s'" % m.commits[max(m.commits)].message if m.commits[max(m.commits)].message else "")) + maxi = max(m.commits) if m.commits else m.branches[branch.number].revision + printo(" %s b%02d%s @%s (%s) with %d commits%s" % ("*" if current == branch.number else " ", branch.number, ((" %%%ds" % (sl + 2)) % ("'%s'" % branch.name)) if branch.name else "", strftime(branch.ctime), "in sync" if branch.inSync else "dirty", len(m.commits), ". Last comment: '%s'" % m.commits[maxi].message if maxi in m.commits and m.commits[maxi].message else "")) if m.track or m.picky and (len(m.branches[m.branch].tracked) > 0 or len(m.branches[m.branch].untracked) > 0): info("\nTracked file patterns:") # TODO print matching untracking patterns side-by-side printo(ajoin(" | ", m.branches[m.branch].tracked, "\n")) @@ -585,13 +624,14 @@ def exitOnChanges(argument:str? = None, options:str[] = [], check:bool = True, c branch, revision = m.parseRevisionString(argument) # for early abort if branch is None: Exit("Branch '%s' doesn't exist. Cannot proceed" % argument) m.loadBranch(m.branch) # knows last commits of *current* branch + maxi:int = max(m.commits) if m.commits else m.branches[m.branch].revision # Determine current changes trackingPatterns:FrozenSet[str] = m.getTrackingPatterns() untrackingPatterns:FrozenSet[str] = m.getTrackingPatterns(negative = True) - m.computeSequentialPathSet(m.branch, max(m.commits)) # load all commits up to specified revision + m.computeSequentialPathSet(m.branch, maxi) # load all commits up to specified revision changes, msg = m.findChanges( - m.branch if commit else None, max(m.commits) + 1 if commit else None, checkContent = strict, + m.branch if commit else None, maxi + 1 if commit else None, checkContent = strict, considerOnly = onlys if not (m.track or m.picky) else conditionalIntersection(onlys, trackingPatterns), dontConsider = excps if not (m.track or m.picky) else excps ?? untrackingPatterns, progress = '--progress' in options) @@ -605,10 +645,11 @@ def exitOnChanges(argument:str? = None, options:str[] = [], check:bool = True, c if argument is not None: # branch/revision specified m.loadBranch(branch) # knows commits of target branch + maxi = max(m.commits) if m.commits else m.branches[m.branch].revision revision = revision if revision >= 0 else len(m.commits) + revision # negative indexing - if revision < 0 or revision > max(m.commits): Exit("Unknown revision r%02d" % revision) + if revision < 0 or revision > maxi: Exit("Unknown revision r%02d" % revision) return (m, branch, revision, changes, strict, force, m.getTrackingPatterns(branch), m.getTrackingPatterns(branch, negative = True)) - (m, m.branch, max(m.commits) + (1 if commit else 0), changes, strict, force, trackingPatterns, untrackingPatterns) + (m, m.branch, maxi + (1 if commit else 0), changes, strict, force, trackingPatterns, untrackingPatterns) def switch(argument:str, options:List[str] = [], onlys:FrozenSet[str]? = None, excps:FrozenSet[str]? = None): ''' Continue work on another branch, replacing file tree changes. ''' @@ -716,15 +757,15 @@ def update(argument:str, options:str[] = [], onlys:FrozenSet[str]? = None, excps m.branch = currentBranch # need to restore setting before saving TODO operate on different objects instead m.saveBranches() -def delete(argument:str, options:str[] = []): +def destroy(argument:str, options:str[] = []): ''' Remove a branch entirely. ''' m, branch, revision, changes, strict, force, trackingPatterns, untrackingPatterns = exitOnChanges(None, options) if len(m.branches) == 1: Exit("Cannot remove the only remaining branch. Use 'sos online' to leave offline mode") branch, revision = m.parseRevisionString(argument) # not from exitOnChanges, because we have to set argument to None there if branch is None or branch not in m.branches: Exit("Cannot delete unknown branch %r" % branch) - debug(MARKER + "Removing branch %d%s..." % (branch, " '%s'" % m.branches[branch].name if m.branches[branch].name else "")) + debug(MARKER + "Removing branch b%02d%s..." % (branch, " '%s'" % (m.branches[branch].name ?? ""))) binfo = m.removeBranch(branch) # need to keep a reference to removed entry for output below - info(MARKER + "Branch b%02d%s removed" % (branch, " '%s'" % binfo.name if binfo.name else "")) + info(MARKER + "Branch b%02d%s removed" % (branch, " '%s'" % (binfo.name ?? ""))) def add(relPath:str, pattern:str, options:str[] = [], negative:bool = False): ''' Add a tracked files pattern to current branch's tracked files. negative means tracking blacklisting. ''' @@ -784,27 +825,32 @@ def ls(folder:str? = None, options:str[] = []): if not ignore: for pattern in (p for p in trackingPatterns if os.path.dirname(p).replace(os.sep, SLASH) == relPath): # only patterns matching current folder if fnmatch.fnmatch(file, os.path.basename(pattern)): matches.append(os.path.basename(pattern)) - matches.sort(key = (element) -> len(element)) + matches.sort(key = (element) -> len(element)) # sort in-place printo("%s %s%s" % ("IGN" if ignore is not None else ("TRK" if len(matches) > 0 else "\u00b7\u00b7\u00b7"), file, " (%s)" % ignore if ignore is not None else (" (%s)" % ("; ".join(matches)) if len(matches) > 0 else ""))) def log(options:str[] = []): ''' List previous commits on current branch. ''' m:Metadata = Metadata() m.loadBranch(m.branch) # knows commit history + maxi:int = max(m.commits) if m.commits else m.branches[m.branch].revision # one commit guaranteed for first offline branch, for fast-branched branches a revision in branchinfo info(MARKER + "Offline commit history of branch '%s'" % m.branches[m.branch].name ?? "r%02d" % m.branch) # TODO also retain info of "from branch/revision" on branching? - nl = len("%d" % max(m.commits)) # determine space needed for revision - changesetIterator:Iterator[Dict[str,PathInfo]]? = m.computeSequentialPathSetIterator(m.branch, max(m.commits)) - maxWidth:int = max([wcswidth(commit.message ?? "") for commit in m.commits.values()]) + nl:int = len("%d" % maxi) # determine space needed for revision + changesetIterator:Iterator[Dict[str,PathInfo]]? = m.computeSequentialPathSetIterator(m.branch, maxi) olds:FrozenSet[str] = f{} # last revision's entries - for no in range(max(m.commits) + 1): - commit:CommitInfo = m.commits[no] + commit:CommitInfo + for no in range(maxi + 1): + if no in m.commits: commit = m.commits[no] + else: # TODO clean this code + n:Metadata = Metadata() + n.loadBranch(n.getParentBranch(m.branch, no)) + commit = n.commits[no] nxts:Dict[str,PathInfo] = next(changesetIterator) news:FrozenSet[str] = frozenset(nxts.keys()) # side-effect: updates m.paths _add:FrozenSet[str] = news - olds _del:FrozenSet[str] = olds - news _mod:FrozenSet[str] = frozenset([_ for _, info in nxts.items() if _ in m.paths and m.paths[_].size != info.size and (m.paths[_].hash != info.hash if m.strict else m.paths[_].mtime != info.mtime)]) _txt:int = len([a for a in _add if m.isTextType(a)]) - printo(" %s r%s @%s (+%02d/-%02d/*%02d +%02dT) |%s|%s" % ("*" if commit.number == max(m.commits) else " ", ("%%%ds" % nl) % commit.number, strftime(commit.ctime), len(_add), len(_del), len(_mod), _txt, (commit.message ?? "").ljust(maxWidth), "TAG" if (commit.message ?? "") in m.tags else "")) + printo(" %s r%s @%s (+%02d/-%02d/*%02d +%02dT) |%s|%s" % ("*" if commit.number == maxi else " ", ("%%%ds" % nl) % commit.number, strftime(commit.ctime), len(_add), len(_del), len(_mod), _txt, (commit.message ?? ""), "TAG" if (commit.message ?? "") in m.tags else "")) if '--changes' in options: m.listChanges(ChangeSet({a: None for a in _add}, {d: None for d in _del}, {m: None for m in _mod}, {})) # TODO moves detection? if '--diff' in options: pass # _diff(m, changes) # needs from revision diff olds = news @@ -813,12 +859,12 @@ def dump(argument:str, options:str[] = []): ''' Exported entire repository as archive for easy transfer. ''' debug(MARKER + "Dumping repository to archive...") progress:bool = '--progress' in options - import zipfile # TODO display compression ratio (if any) - try: import zlib; compression = zipfile.ZIP_DEFLATED + import zipfile + try: import zlib; compression = zipfile.ZIP_DEFLATED # HINT zlib is the library that contains the deflated algorithm except: compression = zipfile.ZIP_STORED if argument is None: Exit("Argument missing (target filename)") - argument = argument if "." in argument else argument + ".sos.zip" + argument = argument if "." in argument else argument + DUMP_FILE # TODO this logic lacks a bit, "v1.2" would not receive the suffix if os.path.exists(encode(argument)): try: shutil.copy2(encode(argument), encode(argument + BACKUP_SUFFIX)) except Exception as E: Exit("Error creating backup copy before dumping. Please resolve and retry. %r" % E) @@ -828,6 +874,7 @@ def dump(argument:str, options:str[] = []): totalsize:int = 0 start_time:float = time.time() for dirpath, dirnames, filenames in os.walk(repopath): # TODO use index knowledge instead of walking to avoid adding stuff not needed? + printo(dirpath.ljust(termWidth)) # TODO improve progress indicator output to | dir | dumpuing file dirpath = decode(dirpath) dirnames[:] = [decode(d) for d in dirnames] filenames[:] = [decode(f) for f in filenames] @@ -838,7 +885,7 @@ def dump(argument:str, options:str[] = []): show:str? = indicator.getIndicator() if progress else None if show: printo(("\rDumping %s @%.2f MiB/s %s" % (show, totalsize / (MEBI * (time.time() - start_time)), filename)).ljust(termWidth), nl = "") _zip.write(abspath, relpath.replace(os.sep, "/")) # write entry into archive - info("\r" + (MARKER + "Finished dumping entire repository.").ljust(termWidth)) # clean line + info("\r" + (MARKER + "Finished dumping entire repository @%.2f MiB/s." % (totalsize / (MEBI * (time.time() - start_time)))).ljust(termWidth)) # clean line def config(arguments:List[str], options:List[str] = []): command, key, value = (arguments + [None] * 2)[:3] @@ -897,7 +944,7 @@ def move(relPath:str, pattern:str, newRelPath:str, newPattern:str, options:List[ ''' Path differs: Move files, create folder if not existing. Pattern differs: Attempt to rename file, unless exists in target or not unique. for "mvnot" don't do any renaming (or do?) ''' - # TODO info(MARKER + TODO write tests for that + debug(MARKER + "Renaming %r to %r" % (pattern, newPattern)) force:bool = '--force' in options soft:bool = '--soft' in options if not os.path.exists(encode(relPath.replace(SLASH, os.sep))) and not force: Exit("Source folder doesn't exist. Use --force to proceed anyway") @@ -935,49 +982,49 @@ def move(relPath:str, pattern:str, newRelPath:str, newPattern:str, options:List[ patterns[patterns.index(pattern)] = newPattern m.saveBranches() -def parse(root:str, cwd:str): - ''' Main operation. Main has already chdir into VCS root folder, cwd is original working directory for add, rm. ''' +def parse(root:str, cwd:str, cmd:str): + ''' Main operation. Main has already chdir into VCS root folder, cwd is original working directory for add, rm, mv. ''' debug("Parsing command-line arguments...") try: command = sys.argv[1].strip() if len(sys.argv) > 1 else "" - arguments:Union[List[str],str,None] = [c.strip() for c in sys.argv[2:] if not c.startswith("--")] - if len(arguments) == 0: arguments = [None] + arguments:List[str?] = [c.strip() for c in sys.argv[2:] if not c.startswith("--")] options = [c.strip() for c in sys.argv[2:] if c.startswith("--")] onlys, excps = parseOnlyOptions(cwd, options) # extracts folder-relative information for changes, commit, diff, switch, update - debug("Processing command %r with arguments %r and options %r." % (command ?? "", arguments if arguments else "", options)) - if command[:1] in "amr": relPath, pattern = relativize(root, os.path.join(cwd, arguments[0] if arguments else ".")) + debug("Processing command %r with arguments %r and options %r." % (command, [_ for _ in arguments if _ is not None], options)) + if command[:1] in "amr": relPath, pattern = relativize(root, os.path.join(cwd, arguments[0] if arguments else "." )) if command[:1] == "m": - if len(arguments) < 2: Exit("Need a second file pattern argument as target for move or config command") - newRelPath, newPattern = relativize(root, os.path.join(cwd, options[0])) - if command[:1] == "a": add(relPath, pattern, options, negative = "n" in command) # also addnot - elif command[:1] == "b": branch(arguments[0], options) + if len(arguments) < 2: Exit("Need a second file pattern argument as target for move command") + newRelPath, newPattern = relativize(root, os.path.join(cwd, arguments[1])) + arguments[:] = (arguments + [None] * 3)[:3] + if command[:1] == "a": add(relPath, pattern, options, negative = "n" in command) # addnot + elif command[:1] == "b": branch(arguments[0], arguments[1], options) elif command[:3] == "com": commit(arguments[0], options, onlys, excps) elif command[:2] == "ch": changes(arguments[0], options, onlys, excps) # "changes" (legacy) elif command[:2] == "ci": commit(arguments[0], options, onlys, excps) elif command[:3] == 'con': config(arguments, options) - elif command[:2] == "de": delete(arguments[0], options) + elif command[:2] == "de": destroy(arguments[0], options) elif command[:2] == "di": diff(arguments[0], options, onlys, excps) elif command[:2] == "du": dump(arguments[0], options) elif command[:1] == "h": usage(APPNAME, version.__version__) elif command[:2] == "lo": log(options) elif command[:2] == "li": ls(os.path.relpath(arguments[0] ?? cwd, root), options) - elif command[:2] == "ls": ls(os.path.relpath(arguments[0] ?? cwd, root), options) # TODO avoid and/or normalize root super paths (..)? - elif command[:1] == "m": move(relPath, pattern, newRelPath, newPattern, options[1:], negative = "n" in command) # also mvnot - elif command[:2] == "of": offline(arguments[0], options) + elif command[:2] == "ls": ls(os.path.relpath(arguments[0] ?? cwd, root), options) + elif command[:1] == "m": move(relPath, pattern, newRelPath, newPattern, options, negative = "n" in command) # mvnot + elif command[:2] == "of": offline(arguments[0], arguments[1], options) elif command[:2] == "on": online(options) - elif command[:1] == "r": remove(relPath, pattern, negative = "n" in command) # also rmnot - elif command[:2] == "st": changes(arguments[0], options, onlys, excps) + elif command[:1] == "r": remove(relPath, pattern, negative = "n" in command) # rmnot + elif command[:2] == "st": changes(arguments[0], options, onlys, excps, useChanges = cmd == "fossil") elif command[:2] == "sw": switch(arguments[0], options, onlys, excps) elif command[:1] == "u": update(arguments[0], options, onlys, excps) elif command[:1] == "v": usage(APPNAME, version.__version__, short = True) else: Exit("Unknown command '%s'" % command) - Exit(code = 0) + Exit(code = 0) # regular exit except Exception, RuntimeError as E: exception(E) - Exit("An internal error occurred in SOS. Please report above message to the project maintainer at https://github.com/ArneBachmann/sos/issues via 'New Issue'.\nPlease state your installed version via 'sos version', and what you were doing.") + Exit("An internal error occurred in SOS. Please report above message to the project maintainer at https://github.com/ArneBachmann/sos/issues via 'New Issue'.\nPlease state your installed version via 'sos version', and what you were doing") def main(): - global debug, info, warn, error + global debug, info, warn, error # to modify logger logging.basicConfig(level = level, stream = sys.stderr, format = ("%(asctime)-23s %(levelname)-8s %(name)s:%(lineno)d | %(message)s" if '--log' in sys.argv else "%(message)s")) _log = Logger(logging.getLogger(__name__)); debug, info, warn, error = _log.debug, _log.info, _log.warn, _log.error for option in (o for o in ['--log', '--verbose', '-v', '--sos'] if o in sys.argv): sys.argv.remove(option) # clean up program arguments @@ -986,12 +1033,13 @@ def main(): root, vcs, cmd = findSosVcsBase() # root is None if no .sos folder exists up the folder tree (still working online); vcs is checkout/repo root folder; cmd is the VCS base command debug("Found root folders for SOS|VCS: %s|%s" % (root ?? "-", vcs ?? "-")) defaults["defaultbranch"] = vcsBranches.get(cmd, "trunk") ?? "default" # sets dynamic default with SVN fallback - if force_sos or root is not None or (command ?? "")[:2] == "of" or (command ?? "")[:1] in ["h", "v"]: # in offline mode or just going offline TODO what about git config? + defaults["useChangesCommand"] = cmd == "fossil" # sets dynamic default with SVN fallback + if force_sos or root is not None or (command ?? "")[:2] == "of" or (command ?? "")[:1] in "hv": # in offline mode or just going offline TODO what about git config? cwd = os.getcwd() os.chdir(cwd if command[:2] == "of" else root ?? cwd) - parse(root, cwd) + parse(root, cwd, cmd) elif force_vcs or cmd is not None: # online mode - delegate to VCS - info("SOS: Running '%s %s'" % (cmd, " ".join(sys.argv[1:]))) + info("%s: Running '%s %s'" % (COMMAND.upper(), cmd, " ".join(sys.argv[1:]))) import subprocess # only required in this section process = subprocess.Popen([cmd] + sys.argv[1:], shell = False, stdin = subprocess.PIPE, stdout = sys.stdout, stderr = sys.stderr) inp:str = "" diff --git a/sos/tests.coco b/sos/tests.coco index aa1f592..4cebd19 100644 --- a/sos/tests.coco +++ b/sos/tests.coco @@ -5,8 +5,6 @@ import builtins, codecs, enum, json, logging, os, shutil, sys, time, traceback, from io import BytesIO, BufferedRandom, TextIOWrapper try: from unittest import mock # Python 3 except: import mock # installed via pip -try: from typing import Any, List, Union # only required for mypy -except: pass testFolder = os.path.abspath(os.path.join(os.getcwd(), "test")) import configr # optional dependency @@ -92,8 +90,9 @@ class Tests(unittest.TestCase): os.chdir(testFolder) - def assertAllIn(_, what:str[], where:Union[str,List[str]]): + def assertAllIn(_, what:str[], where:Union[str,List[str]], only:bool = False): for w in what: _.assertIn(w, where) + if only: _.assertEqual(len(what), len(where)) def assertAllNotIn(_, what:str[], where:Union[str,List[str]]): for w in what: _.assertNotIn(w, where) @@ -154,8 +153,8 @@ class Tests(unittest.TestCase): _.assertEqual(0, len(changes.deletions)) _.assertEqual(1, len(changes.modifications)) _.assertEqual(0, len(changes.moves)) - _.assertTrue(os.path.exists(sos.branchFolder(0, 1))) - _.assertTrue(os.path.exists(sos.branchFolder(0, 1, file = "03b69bc801ae11f1ff2a71a50f165996d0ad681b4f822df13329a27e53f0fcd2"))) + _.assertTrue(os.path.exists(sos.revisionFolder(0, 1))) + _.assertTrue(os.path.exists(sos.revisionFolder(0, 1, file = "03b69bc801ae11f1ff2a71a50f165996d0ad681b4f822df13329a27e53f0fcd2"))) # TODO test moves def testPatternPaths(_): @@ -164,20 +163,48 @@ class Tests(unittest.TestCase): _.createFile("sub" + os.sep + "file1", "sdfsdf") sos.add("sub", "sub/file?") sos.commit("test") # should pick up sub/file1 pattern - _.assertEqual(2, len(os.listdir(os.path.join(sos.metaFolder, "b0", "r1")))) # sub/file1 was added + _.assertEqual(2, len(os.listdir(sos.revisionFolder(0, 1)))) # sub/file1 was added _.createFile(1) try: sos.commit("nothing"); _.fail() # should not commit anything, as the file in base folder doesn't match the tracked pattern except: pass + def testNoArgs(_): + pass # call "sos" without arguments should simply show help or info about missing arguments + def testAutoUpgrade(_): sos.offline() with codecs.open(sos.encode(os.path.join(sos.metaFolder, sos.metaFile)), "r", encoding = sos.UTF8) as fd: repo, branches, config = json.load(fd) - del repo["format"] # simulate pre-1.4 - repo["version"] = "0" - branches[:] = [branch[:-1] for branch in branches] + repo["version"] = "0" # lower than any pip version + branches[:] = [branch[:5] for branch in branches] # simulate some older state + del repo["format"] # simulate pre-1.3.5 with codecs.open(sos.encode(os.path.join(sos.metaFolder, sos.metaFile)), "w", encoding = sos.UTF8) as fd: json.dump((repo, branches, config), fd, ensure_ascii = False) out = wrapChannels(() -> sos.status(options = ["--repo"])) - _.assertAllIn(["Upgraded repository metadata to match SOS version '2018.1210.3028'", "Upgraded repository metadata to match SOS version '1.4'"], out) + _.assertAllIn(["Upgraded repository metadata to match SOS version '2018.1210.3028'", "Upgraded repository metadata to match SOS version '1.3.5'"], out) + + def testFastBranching(_): + _.createFile(1) + sos.offline(options = ["--strict"]) # b0/r0 = ./file1 + _.createFile(2) + os.unlink("file1") + sos.commit() # b0/r1 = ./file2 + sos.branch(options = ["--fast", "--last"]) # branch b1 from b0/1 TODO modify once --fast becomes the new normal + _.assertAllIn([sos.metaFile, sos.metaBack, "b0", "b1"], os.listdir(sos.metaFolder), only = True) + _.createFile(3) + sos.commit() # b1/r2 = ./file2, ./file3 + _.assertAllIn([sos.metaFile, "r2"], os.listdir(sos.branchFolder(1)), only = True) + sos.branch(options = ["--fast", "--last"]) # branch b2 from b1/2 + sos.destroy("0") # remove parent of b1 and transitive parent of b2 + _.assertAllIn([sos.metaFile, sos.metaBack, "b0_last", "b1", "b2"], os.listdir(sos.metaFolder), only = True) # branch 0 was removed + _.assertAllIn([sos.metaFile, "r0", "r1", "r2"], os.listdir(sos.branchFolder(1)), only = True) # revisions were copied to branch 1 + _.assertAllIn([sos.metaFile, "r0", "r1", "r2"], os.listdir(sos.branchFolder(2)), only = True) # revisions were copied to branch 1 + # TODO test also other functions like status --repo, log + + def testGetParentBranch(_): + m = sos.Accessor({"branches": {0: sos.Accessor({"parent": None, "revision": None}), 1: sos.Accessor({"parent": 0, "revision": 1})}}) + _.assertEqual(0, sos.Metadata.getParentBranch(m, 1, 0)) + _.assertEqual(0, sos.Metadata.getParentBranch(m, 1, 1)) + _.assertEqual(1, sos.Metadata.getParentBranch(m, 1, 2)) + _.assertEqual(0, sos.Metadata.getParentBranch(m, 0, 10)) def testTokenizeGlobPattern(_): _.assertEqual([], sos.tokenizeGlobPattern("")) @@ -236,11 +263,11 @@ class Tests(unittest.TestCase): _.existsFile(1, "something else") def testComputeSequentialPathSet(_): - os.makedirs(sos.branchFolder(0, 0)) - os.makedirs(sos.branchFolder(0, 1)) - os.makedirs(sos.branchFolder(0, 2)) - os.makedirs(sos.branchFolder(0, 3)) - os.makedirs(sos.branchFolder(0, 4)) + os.makedirs(sos.revisionFolder(0, 0)) + os.makedirs(sos.revisionFolder(0, 1)) + os.makedirs(sos.revisionFolder(0, 2)) + os.makedirs(sos.revisionFolder(0, 3)) + os.makedirs(sos.revisionFolder(0, 4)) m = sos.Metadata(os.getcwd()) m.branch = 0 m.commit = 2 @@ -257,6 +284,8 @@ class Tests(unittest.TestCase): m.saveCommit(0, 4) # readd m.commits = {i: sos.CommitInfo(i, 0, None) for i in range(5)} m.saveBranch(0) + m.branches = {0: sos.BranchInfo(0, 0), 1: sos.BranchInfo(1, 0)} + m.saveBranches() m.computeSequentialPathSet(0, 4) _.assertEqual(2, len(m.paths)) @@ -283,7 +312,7 @@ class Tests(unittest.TestCase): _.assertAllIn(["r0", sos.metaFile], os.listdir("." + os.sep + sos.metaFolder + os.sep + "b0")) _.assertEqual(2, len(os.listdir("." + os.sep + sos.metaFolder))) # only branch folder and meta data file _.assertEqual(2, len(os.listdir("." + os.sep + sos.metaFolder + os.sep + "b0"))) # only commit folder and meta data file - _.assertEqual(1, len(os.listdir(sos.branchFolder(0, 0)))) # only meta data file + _.assertEqual(1, len(os.listdir(sos.revisionFolder(0, 0)))) # only meta data file def testOfflineWithFiles(_): _.createFile(1, "x" * 100) @@ -295,7 +324,7 @@ class Tests(unittest.TestCase): _.assertAllIn([sos.metaFile, "03b69bc801ae11f1ff2a71a50f165996d0ad681b4f822df13329a27e53f0fcd2", "b9ee10a87f612e299a6eb208210bc0898092a64c48091327cc2aaeee9b764ffa"], os.listdir("." + os.sep + sos.metaFolder + os.sep + "b0" + os.sep + "r0")) _.assertEqual(2, len(os.listdir("." + os.sep + sos.metaFolder))) # only branch folder and meta data file _.assertEqual(2, len(os.listdir("." + os.sep + sos.metaFolder + os.sep + "b0"))) # only commit folder and meta data file - _.assertEqual(3, len(os.listdir(sos.branchFolder(0, 0)))) # only meta data file plus branch base file copies + _.assertEqual(3, len(os.listdir(sos.revisionFolder(0, 0)))) # only meta data file plus branch base file copies def testBranch(_): _.createFile(1, "x" * 100) @@ -305,14 +334,14 @@ class Tests(unittest.TestCase): _.assertAllIn(["b0", "b1", sos.metaFile], os.listdir("." + os.sep + sos.metaFolder)) _.assertEqual(list(sorted(os.listdir("." + os.sep + sos.metaFolder + os.sep + "b0"))), list(sorted(os.listdir("." + os.sep + sos.metaFolder + os.sep + "b1")))) - _.assertEqual(list(sorted(os.listdir(sos.branchFolder(0, 0)))), - list(sorted(os.listdir(sos.branchFolder(1, 0))))) + _.assertEqual(list(sorted(os.listdir(sos.revisionFolder(0, 0)))), + list(sorted(os.listdir(sos.revisionFolder(1, 0))))) _.createFile(1, "z") # modify file sos.branch() # b2/r0 branch to unnamed branch with modified file tree contents _.assertNotEqual(os.stat("." + os.sep + sos.metaFolder + os.sep + "b1" + os.sep + "r0" + os.sep + "b9ee10a87f612e299a6eb208210bc0898092a64c48091327cc2aaeee9b764ffa").st_size, os.stat("." + os.sep + sos.metaFolder + os.sep + "b2" + os.sep + "r0" + os.sep + "b9ee10a87f612e299a6eb208210bc0898092a64c48091327cc2aaeee9b764ffa").st_size) _.createFile(3, "z") - sos.branch("from_last_revision", ["--last", "--stay"]) # b3/r0 create copy of other file1,file2 and don't switch + sos.branch("from_last_revision", options = ["--last", "--stay"]) # b3/r0 create copy of other file1,file2 and don't switch _.assertEqual(list(sorted(os.listdir("." + os.sep + sos.metaFolder + os.sep + "b3" + os.sep + "r0"))), list(sorted(os.listdir("." + os.sep + sos.metaFolder + os.sep + "b2" + os.sep + "r0")))) # Check sos.status output which branch is marked @@ -333,8 +362,8 @@ class Tests(unittest.TestCase): _.assertEqual(1, len(changes.modifications)) sos.commit("message") _.assertAllIn(["r0", "r1", sos.metaFile], os.listdir("." + os.sep + sos.metaFolder + os.sep + "b0")) - _.assertAllIn([sos.metaFile, "b9ee10a87f612e299a6eb208210bc0898092a64c48091327cc2aaeee9b764ffa"], os.listdir(sos.branchFolder(0, 1))) - _.assertEqual(2, len(os.listdir(sos.branchFolder(0, 1)))) # no further files, only the modified one + _.assertAllIn([sos.metaFile, "b9ee10a87f612e299a6eb208210bc0898092a64c48091327cc2aaeee9b764ffa"], os.listdir(sos.revisionFolder(0, 1))) + _.assertEqual(2, len(os.listdir(sos.revisionFolder(0, 1)))) # no further files, only the modified one _.assertEqual(1, len(sos.changes("/0").modifications)) # vs. explicit revision on current branch _.assertEqual(1, len(sos.changes("0/0").modifications)) # vs. explicit branch/revision _.createFile(1, "") # modify to empty file, mentioned in meta data, but not stored as own file @@ -344,7 +373,7 @@ class Tests(unittest.TestCase): _.assertEqual(1, len(changes.deletions)) _.assertEqual(1, len(changes.modifications)) sos.commit("modified") - _.assertEqual(1, len(os.listdir(sos.branchFolder(0, 2)))) # no additional files, only mentions in metadata + _.assertEqual(1, len(os.listdir(sos.revisionFolder(0, 2)))) # no additional files, only mentions in metadata try: sos.commit("nothing"); _.fail() # expecting Exit due to no changes except: pass @@ -425,9 +454,9 @@ class Tests(unittest.TestCase): _.assertTrue(_.existsFile(1)) sos.commit("third", ["--force"]) # force because nothing to commit. should create r2 with same contents as r1, but as differential from r1, not from r0 (= no changes in meta folder) - _.assertTrue(os.path.exists(sos.branchFolder(0, 2))) - _.assertTrue(os.path.exists(sos.branchFolder(0, 2, file = sos.metaFile))) - _.assertEqual(1, len(os.listdir(sos.branchFolder(0, 2)))) # only meta data file, no differential files + _.assertTrue(os.path.exists(sos.revisionFolder(0, 2))) + _.assertTrue(os.path.exists(sos.revisionFolder(0, 2, file = sos.metaFile))) + _.assertEqual(1, len(os.listdir(sos.revisionFolder(0, 2)))) # only meta data file, no differential files sos.update("/1") # do nothing, as nothing has changed _.assertTrue(_.existsFile(1)) @@ -435,7 +464,7 @@ class Tests(unittest.TestCase): _.createFile(2, "y" * 100) # out = wrapChannels(() -> sos.branch("other")) # won't comply as there are changes # _.assertIn("--force", out) - sos.branch("other", ["--force"]) # automatically including file 2 (as we are in simple mode) + sos.branch("other", options = ["--force"]) # automatically including file 2 (as we are in simple mode) _.assertTrue(_.existsFile(2)) sos.update("trunk", ["--add"]) # only add stuff _.assertTrue(_.existsFile(2)) @@ -528,19 +557,19 @@ class Tests(unittest.TestCase): def testPickyMode(_): ''' Confirm that picky mode reset tracked patterns after commits. ''' - sos.offline("trunk", ["--picky"]) + sos.offline("trunk", None, ["--picky"]) changes = sos.changes() _.assertEqual(0, len(changes.additions)) # do not list any existing file as an addition sos.add(".", "./file?", ["--force"]) _.createFile(1, "aa") sos.commit("First") # add one file - _.assertEqual(2, len(os.listdir(sos.branchFolder(0, 1)))) + _.assertEqual(2, len(os.listdir(sos.revisionFolder(0, 1)))) _.createFile(2, "b") try: sos.commit("Second") # add nothing, because picky except: pass sos.add(".", "./file?") sos.commit("Third") - _.assertEqual(2, len(os.listdir(sos.branchFolder(0, 2)))) + _.assertEqual(2, len(os.listdir(sos.revisionFolder(0, 2)))) out = wrapChannels(() -> sos.log([])).replace("\r", "") _.assertIn(" * r2", out) _.createFile(3, prefix = "sub") @@ -552,22 +581,22 @@ class Tests(unittest.TestCase): def testTrackedSubfolder(_): ''' See if patterns for files in sub folders are picked up correctly. ''' os.mkdir("." + os.sep + "sub") - sos.offline("trunk", ["--track"]) + sos.offline("trunk", None, ["--track"]) _.createFile(1, "x") _.createFile(1, "x", prefix = "sub") sos.add(".", "./file?") # add glob pattern to track sos.commit("First") - _.assertEqual(2, len(os.listdir(sos.branchFolder(0, 1)))) # one new file + meta file + _.assertEqual(2, len(os.listdir(sos.revisionFolder(0, 1)))) # one new file + meta file sos.add(".", "sub/file?") # add glob pattern to track sos.commit("Second") # one new file + meta - _.assertEqual(2, len(os.listdir(sos.branchFolder(0, 1)))) # one new file + meta file + _.assertEqual(2, len(os.listdir(sos.revisionFolder(0, 1)))) # one new file + meta file os.unlink("file1") # remove from basefolder _.createFile(2, "y") sos.remove(".", "sub/file?") try: sos.remove(".", "sub/bla"); _.fail() # raises Exit. TODO test the "suggest a pattern" case except: pass sos.commit("Third") - _.assertEqual(2, len(os.listdir(sos.branchFolder(0, 2)))) # one new file + meta + _.assertEqual(2, len(os.listdir(sos.revisionFolder(0, 2)))) # one new file + meta # TODO also check if /file1 and sub/file1 were removed from index def testTrackedMode(_): @@ -581,18 +610,18 @@ class Tests(unittest.TestCase): _.createFile("a123a") # untracked file "a123a" sos.add(".", "./file?") # add glob tracking pattern sos.commit("second") # versions "file1" - _.assertEqual(2, len(os.listdir(sos.branchFolder(0, 1)))) # one new file + meta file + _.assertEqual(2, len(os.listdir(sos.revisionFolder(0, 1)))) # one new file + meta file out = wrapChannels(() -> sos.status()).replace("\r", "") _.assertIn(" | ./file?", out) _.createFile(2) # untracked file "file2" sos.commit("third") # versions "file2" - _.assertEqual(2, len(os.listdir(sos.branchFolder(0, 2)))) # one new file + meta file + _.assertEqual(2, len(os.listdir(sos.revisionFolder(0, 2)))) # one new file + meta file os.mkdir("." + os.sep + "sub") _.createFile(3, prefix = "sub") # untracked file "sub/file3" sos.commit("fourth", ["--force"]) # no tracking pattern matches the subfolder - _.assertEqual(1, len(os.listdir(sos.branchFolder(0, 3)))) # meta file only, no other tracked path/file + _.assertEqual(1, len(os.listdir(sos.revisionFolder(0, 3)))) # meta file only, no other tracked path/file sos.branch("Other") # second branch containing file1 and file2 tracked by "./file?" sos.remove(".", "./file?") # remove tracking pattern, but don't touch previously created and versioned files @@ -603,7 +632,7 @@ class Tests(unittest.TestCase): _.assertEqual(1, len(changes.additions)) # detected one addition "a123a", but won't recognize untracking files as deletion sos.commit("Second_2") - _.assertEqual(2, len(os.listdir(sos.branchFolder(1, 1)))) # "a123a" + meta file + _.assertEqual(2, len(os.listdir(sos.revisionFolder(1, 1)))) # "a123a" + meta file _.existsFile(1, b"x" * 10) _.existsFile(2, b"x" * 10) @@ -617,7 +646,7 @@ class Tests(unittest.TestCase): _.createFile("axxxa") # new file that should be tracked on "test" now that we integrated "Other" sos.commit("fifth") # create new revision after integrating updates from second branch - _.assertEqual(3, len(os.listdir(sos.branchFolder(0, 4)))) # one new file from other branch + one new in current folder + meta file + _.assertEqual(3, len(os.listdir(sos.revisionFolder(0, 4)))) # one new file from other branch + one new in current folder + meta file sos.switch("Other") # switch back to just integrated branch that tracks only "a*a" - shouldn't do anything _.assertTrue(os.path.exists("." + os.sep + "file1")) _.assertTrue(os.path.exists("." + os.sep + "a123a")) @@ -652,13 +681,13 @@ class Tests(unittest.TestCase): sos.offline("master", options = ["--force"]) out = wrapChannels(() -> sos.changes(options = ['--progress'])).replace("\r", "").split("\n") _.assertFalse(any("Compression advantage" in line for line in out)) # simple mode should always print this to stdout - _.assertTrue(_.existsFile(sos.branchFolder(0, 0, file = "b9ee10a87f612e299a6eb208210bc0898092a64c48091327cc2aaeee9b764ffa"), b"x" * 10)) + _.assertTrue(_.existsFile(sos.revisionFolder(0, 0, file = "b9ee10a87f612e299a6eb208210bc0898092a64c48091327cc2aaeee9b764ffa"), b"x" * 10)) setRepoFlag("compress", True) # was plain = uncompressed before _.createFile(2) out = wrapChannels(() -> sos.commit("Added file2", options = ['--progress'])).replace("\r", "").split("\n") _.assertTrue(any("Compression advantage" in line for line in out)) - _.assertTrue(_.existsFile(sos.branchFolder(0, 1, file = "03b69bc801ae11f1ff2a71a50f165996d0ad681b4f822df13329a27e53f0fcd2"))) # exists - _.assertFalse(_.existsFile(sos.branchFolder(0, 1, file = "03b69bc801ae11f1ff2a71a50f165996d0ad681b4f822df13329a27e53f0fcd2"), b"x" * 10)) # but is compressed instead + _.assertTrue(_.existsFile(sos.revisionFolder(0, 1, file = "03b69bc801ae11f1ff2a71a50f165996d0ad681b4f822df13329a27e53f0fcd2"))) # exists + _.assertFalse(_.existsFile(sos.revisionFolder(0, 1, file = "03b69bc801ae11f1ff2a71a50f165996d0ad681b4f822df13329a27e53f0fcd2"), b"x" * 10)) # but is compressed instead def testLocalConfig(_): sos.offline("bla", options = []) @@ -759,10 +788,10 @@ class Tests(unittest.TestCase): # TODO test same for simple mode _.createFile(1) sos.defaults.ignores[:] = ["file*"] # replace in-place - sos.offline("xx", ["--track", "--strict"]) # because nothing to commit due to ignore pattern + sos.offline("xx", options = ["--track", "--strict"]) # because nothing to commit due to ignore pattern sos.add(".", "./file*") # add tracking pattern for "file1" sos.commit(options = ["--force"]) # attempt to commit the file - _.assertEqual(1, len(os.listdir(sos.branchFolder(0, 1)))) # only meta data, file1 was ignored + _.assertEqual(1, len(os.listdir(sos.revisionFolder(0, 1)))) # only meta data, file1 was ignored try: sos.online(); _.fail() # Exit because dirty except: pass # exception expected _.createFile("x2") # add another change @@ -774,25 +803,25 @@ class Tests(unittest.TestCase): _.createFile(1) sos.defaults.ignoresWhitelist[:] = ["file*"] - sos.offline("xx", ["--track"]) + sos.offline("xx", None, ["--track"]) sos.add(".", "./file*") sos.commit() # should NOT ask for force here - _.assertEqual(2, len(os.listdir(sos.branchFolder(0, 1)))) # meta data and "file1", file1 was whitelisted + _.assertEqual(2, len(os.listdir(sos.revisionFolder(0, 1)))) # meta data and "file1", file1 was whitelisted def testRemove(_): _.createFile(1, "x" * 100) sos.offline("trunk") - try: sos.delete("trunk"); _fail() + try: sos.destroy("trunk"); _fail() except: pass _.createFile(2, "y" * 10) sos.branch("added") # creates new branch, writes repo metadata, and therefore creates backup copy - sos.delete("trunk") - _.assertEqual(3, len(os.listdir("." + os.sep + sos.metaFolder))) # meta data file, backup and "b1" + sos.destroy("trunk") + _.assertAllIn([sos.metaFile, sos.metaBack, "b0_last", "b1"], os.listdir("." + os.sep + sos.metaFolder)) _.assertTrue(os.path.exists("." + os.sep + sos.metaFolder + os.sep + "b1")) _.assertFalse(os.path.exists("." + os.sep + sos.metaFolder + os.sep + "b0")) sos.branch("next") _.createFile(3, "y" * 10) # make a change - sos.delete("added", "--force") # should succeed + sos.destroy("added", "--force") # should succeed def testUsage(_): try: sos.usage(); _.fail() # TODO expect sys.exit(0) @@ -826,9 +855,9 @@ class Tests(unittest.TestCase): sos.add(".", "./file1") sos.add(".", "./file2") sos.commit(onlys = f{"./file1"}) - _.assertEqual(2, len(os.listdir(sos.branchFolder(0, 1)))) # only meta and file1 + _.assertEqual(2, len(os.listdir(sos.revisionFolder(0, 1)))) # only meta and file1 sos.commit() # adds also file2 - _.assertEqual(2, len(os.listdir(sos.branchFolder(0, 2)))) # only meta and file1 + _.assertEqual(2, len(os.listdir(sos.revisionFolder(0, 2)))) # only meta and file1 _.createFile(1, "cc") # modify both files _.createFile(2, "dd") try: sos.config(["set", "texttype", "file2"]) @@ -907,8 +936,8 @@ class Tests(unittest.TestCase): _.createFile("1a2b") # should not be tracked _.createFile("a1b2") # should be tracked sos.commit() - _.assertEqual(2, len(os.listdir(sos.branchFolder(0, 1)))) - _.assertTrue(os.path.exists(sos.branchFolder(0, 1, file = "93b38f90892eb5c57779ca9c0b6fbdf6774daeee3342f56f3e78eb2fe5336c50"))) # a1b2 + _.assertEqual(2, len(os.listdir(sos.revisionFolder(0, 1)))) + _.assertTrue(os.path.exists(sos.revisionFolder(0, 1, file = "93b38f90892eb5c57779ca9c0b6fbdf6774daeee3342f56f3e78eb2fe5336c50"))) # a1b2 _.createFile("1a1b1") _.createFile("1a1b2") sos.add(".", "?a?b*") @@ -919,8 +948,8 @@ class Tests(unittest.TestCase): def testHashCollision(_): sos.offline() _.createFile(1) - os.mkdir(sos.branchFolder(0, 1)) - _.createFile("b9ee10a87f612e299a6eb208210bc0898092a64c48091327cc2aaeee9b764ffa", prefix = sos.branchFolder(0, 1)) + os.mkdir(sos.revisionFolder(0, 1)) + _.createFile("b9ee10a87f612e299a6eb208210bc0898092a64c48091327cc2aaeee9b764ffa", prefix = sos.revisionFolder(0, 1)) _.createFile(1) try: sos.commit(); _.fail() # should exit with error due to collision detection except SystemExit as E: _.assertEqual(1, E.code) # TODO will capture exit(0) which is wrong, change to check code in all places @@ -943,6 +972,8 @@ class Tests(unittest.TestCase): # TODO test +++ --- in diff # TODO test +01/-02/*.. + # TODO tests for loadcommit redirection + # TODO test wrong branch/revision after fast branching, would raise exception for -1 otherwise if __name__ == '__main__': logging.basicConfig(level = logging.DEBUG if '-v' in sys.argv or "true" in [os.getenv("DEBUG", "false").strip().lower(), os.getenv("CI", "false").strip().lower()] else logging.INFO, diff --git a/sos/usage.coco b/sos/usage.coco index 992c3e4..ba2f7b2 100644 --- a/sos/usage.coco +++ b/sos/usage.coco @@ -1,5 +1,6 @@ import sys +COMMAND:str = "sos" MARKER:str = r"/SOS/ " @@ -13,7 +14,7 @@ Usage: {cmd} [] [, ...] When operating in of {cmd} When operating in online mode, automatic passthrough to traditional VCS Repository handling: - offline [] Start working offline, creating a branch (named ), default name depending on VCS + offline [ []] Start working offline, creating a branch (named ), default name depending on VCS --compress Compress versioned files (same as `sos config set compress on && sos offline`) --track Setup SVN-style mode: users add/remove tracking patterns per branch --picky Setup Git-style mode: users pick files for each operation @@ -22,9 +23,10 @@ Usage: {cmd} [] [, ...] When operating in of dump [/] Dump entire repository into an archive file Working with branches: - branch [] Create a new branch from current file tree and switch to it + branch [ []] Create a new branch from current file tree and switch to it --last Use last revision, not current file tree, but keep file tree unchanged --stay Don't switch to new branch, continue on current one + --fast Using the reference branching model (experimental) destroy [] Remove (current or specified) branch entirely switch [][/] Continue work on another branch --meta Only switch file tracking patterns for current branch, don't update any files @@ -39,6 +41,8 @@ Usage: {cmd} [] [, ...] When operating in of --tag Memorizes commit message as a tag that can be used instead of numeric revisions diff [][/] List changes in file tree (or `--from` specified revision) vs. last (or specified) revision --to=branch/revision Take "to" revision as target to compare against (instead of current file tree state) + --ignore-whitespace | --iw Ignore white spaces during comparison + --wrap Wrap text around terminal size instead of shortening ls [] [--patterns|--tags] List file tree and mark changes and tracking status Defining file patterns: @@ -86,5 +90,5 @@ Usage: {cmd} [] [, ...] When operating in of --except Avoid operation for specified pattern(s). Available for "changes", "commit", "diff", "switch", and "update" --{cmd} When executing {CMD} not being offline, pass arguments to {CMD} instead (e.g. {cmd} --{cmd} config set key value.) --log Enable logging details - --verbose Enable verbose output, including show compression ratios""".format(appname = appname, cmd = "sos", CMD = "SOS")) + --verbose Enable verbose output, including show compression ratios""".format(appname = appname, cmd = COMMAND, CMD = COMMAND.upper())) sys.exit(0) diff --git a/sos/utility.coco b/sos/utility.coco index 91b4606..9a41a33 100644 --- a/sos/utility.coco +++ b/sos/utility.coco @@ -5,11 +5,8 @@ import bz2, codecs, difflib, hashlib, logging, os, re, sys, time; START_TIME = time.time() try: import enum except: raise Exception("Please install SOS via 'pip install -U sos-vcs[backport]' to get enum support for Python versions prior 3.4") -try: # TODO remove as soon as Coconut supports it - from typing import Any, Callable, Dict, FrozenSet, Generic, IO, List, Sequence, Set, Tuple, Type, TypeVar, Union # only required for mypy - Number = TypeVar("Number", int, float) - DataType = TypeVar("DataType", BranchInfo, ChangeSet, MergeBlock, PathInfo) -except: pass # typing not available (prior to Python 3.5) +from typing import Generic, TypeVar +Number = TypeVar("Number", int, float) try: import wcwidth except: pass # optional dependency @@ -33,6 +30,7 @@ class ProgressIndicator(Counter): ''' Manages a rotating progress indicator. ''' def __init__(_, callback:Optional[(str) -> None] = None): super(ProgressIndicator, _).__init__(-1); _.timer:float = time.time(); _.callback:Optional[(str) -> None] = callback def getIndicator(_) -> str? = + ''' Returns a value only if a certain time has passed. ''' newtime = time.time() if newtime - _.timer < .1: return None _.timer = newtime @@ -51,25 +49,35 @@ class Logger: # Constants _log = Logger(logging.getLogger(__name__)); debug, info, warn, error = _log.debug, _log.info, _log.warn, _log.error -UTF8 = "utf_8" # early used constant, not defined in standard library -CONFIGURABLE_FLAGS = ["strict", "track", "picky", "compress"] -CONFIGURABLE_LISTS = ["texttype", "bintype", "ignores", "ignoreDirs", "ignoresWhitelist", "ignoreDirsWhitelist"] -GLOBAL_LISTS = ["ignores", "ignoreDirs", "ignoresWhitelist", "ignoreDirsWhitelist"] -TRUTH_VALUES = ["true", "yes", "on", "1", "enable", "enabled"] # all lower-case normalized -FALSE_VALUES = ["false", "no", "off", "0", "disable", "disabled"] -PROGRESS_MARKER = ["|", "/", "-", "\\"] -BACKUP_SUFFIX = "_last" +CONFIGURABLE_FLAGS:List[str] = ["strict", "track", "picky", "compress", "useChangesCommand"] +CONFIGURABLE_LISTS:List[str] = ["texttype", "bintype", "ignores", "ignoreDirs", "ignoresWhitelist", "ignoreDirsWhitelist"] +GLOBAL_LISTS:List[str] = ["ignores", "ignoreDirs", "ignoresWhitelist", "ignoreDirsWhitelist"] +TRUTH_VALUES:List[str] = ["true", "yes", "on", "1", "enable", "enabled"] # all lower-case normalized +FALSE_VALUES:List[str] = ["false", "no", "off", "0", "disable", "disabled"] +PROGRESS_MARKER:List[str] = ["|", "/", "-", "\\"] +BACKUP_SUFFIX:str = "_last" metaFolder:str = ".sos" +DUMP_FILE:str = metaFolder + ".zip" metaFile:str = ".meta" metaBack:str = metaFile + BACKUP_SUFFIX -bufSize:int = 1 << 20 # 1 MiB -SVN = "svn" -SLASH = "/" -MEBI = 1 << 20 +MEBI:int = 1 << 20 +bufSize:int = MEBI +UTF8:str = "utf_8" # early used constant, not defined in standard library +SVN:str = "svn" +SLASH:str = "/" +METADATA_FORMAT:int = 1 # counter for incompatible consecutive formats vcsFolders:Dict[str,str] = {".svn": SVN, ".git": "git", ".bzr": "bzr", ".hg": "hg", ".fslckout": "fossil", "_FOSSIL_": "fossil", ".CVS": "cvs"} vcsBranches:Dict[str,str?] = {SVN: "trunk", "git": "master", "bzr": "trunk", "hg": "default", "fossil": None, "cvs": None} NL_NAMES:Dict[bytes,str] = {None: "", b"\r\n": "", b"\n\r": "", b"\n": "", b"\r": ""} -lateBinding:Accessor = Accessor({"verbose": False, "start": 0}) +defaults:Accessor = Accessor({ + "strict": False, "track": False, "picky": False, "compress": False, "useChangesCommand": False, + "texttype": ["*.md", "*.coco", "*.py", "*.pyi", "*.pth"], + "bintype": [], + "ignoreDirs": [".*", "__pycache__", ".mypy_cache"], + "ignoreDirsWhitelist": [], + "ignores": ["__coconut__.py", "*.bak", "*.py[cdo]", "*.class", ".fslckout", "_FOSSIL_", "*%s" % DUMP_FILE], + "ignoresWhitelist": [] + }) # Enums @@ -78,7 +86,7 @@ MergeBlockType = enum.Enum("MergeBlockType", "KEEP INSERT REMOVE REPLACE MOVE") # Value types -data BranchInfo(number:int, ctime:int, name:str? = None, inSync:bool = False, tracked:List[str] = [], untracked:List[str] = [], parent:int? = None) # tracked is a list on purpose, as serialization to JSON needs effort and frequent access is not taking place +data BranchInfo(number:int, ctime:int, name:str? = None, inSync:bool = False, tracked:List[str] = [], untracked:List[str] = [], parent:int? = None, revision:int? = None) # tracked is a list on purpose, as serialization to JSON needs effort and frequent access is not taking place data CommitInfo(number:int, ctime:int, message:str? = None) data PathInfo(nameHash:str, size:int?, mtime:int, hash:str?) # size == None means deleted in this revision data ChangeSet(additions:Dict[str,PathInfo], deletions:Dict[str,PathInfo], modifications:Dict[str,PathInfo], moves:Dict[str,Tuple[str,PathInfo]]) # avoid default assignment of {} as it leads to runtime errors (contains data on init for unknown reason) @@ -86,11 +94,12 @@ data Range(tipe:MergeBlockType, indexes:int[]) # MergeBlockType[1,2,4], line nu data MergeBlock(tipe:MergeBlockType, lines:List[str], line:int, replaces:MergeBlock? = None, changes:Range? = None) data GlobBlock(isLiteral:bool, content:str, index:int) # for file pattern rename/move matching data GlobBlock2(isLiteral:bool, content:str, matches:str) # matching file pattern and input filename for translation +DataType = TypeVar("DataType", BranchInfo, ChangeSet, MergeBlock, PathInfo) # Functions -def printo(s:str, nl:str = "\n"): sys.stdout.buffer.write((s + nl).encode(sys.stdout.encoding, 'backslashreplace')); sys.stdout.flush() # PEP528 compatibility -def printe(s:str, nl:str = "\n"): sys.stderr.buffer.write((s + nl).encode(sys.stderr.encoding, 'backslashreplace')); sys.stderr.flush() +def printo(s:str = "", nl:str = "\n"): sys.stdout.buffer.write((s + nl).encode(sys.stdout.encoding, 'backslashreplace')); sys.stdout.flush() # PEP528 compatibility +def printe(s:str = "", nl:str = "\n"): sys.stderr.buffer.write((s + nl).encode(sys.stderr.encoding, 'backslashreplace')); sys.stderr.flush() def encode(s:str) -> bytes: return os.fsencode(s) # for py->os access of writing filenames # PEP 529 compatibility def decode(b:bytes) -> str: return os.fsdecode(b) # for os->py access of reading filenames try: @@ -164,7 +173,9 @@ def getTermWidth() -> int = except: return 80 termwidth.getTermWidth()[0] -def branchFolder(branch:int, revision:int, base = ".", file = None) -> str = os.path.join(base, metaFolder, "b%d" % branch, "r%d" % revision) + ((os.sep + file) if file else "") +def branchFolder(branch:int, base:str? = None, file:str? = None) -> str = os.path.join(base ?? os.getcwd(), metaFolder, "b%d" % branch) + ((os.sep + file) if file else "") + +def revisionFolder(branch:int, revision:int, base:str? = None, file:str? = None) -> str = os.path.join(branchFolder(branch, base), "r%d" % revision) + ((os.sep + file) if file else "") def Exit(message:str = "", code = 1): printe("[EXIT%s]" % (" %.1fs" % (time.time() - START_TIME) if verbose else "") + (" " + message + "." if message != "" else "")); sys.exit(code) @@ -195,34 +206,39 @@ def hashFile(path:str, compress:bool, saveTo:str? = None, callback:Optional[(str (_hash.hexdigest(), wsize) def getAnyOfMap(map:Dict[str,Any], params:str[], default:Any = None) -> Any = - ''' Utility. ''' + ''' Utility to find any entries of a dictionary in a list to return the dictionaries value. ''' for k, v in map.items(): if k in params: return v default def strftime(timestamp:int? = None) -> str = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(timestamp / 1000. if timestamp is not None else None)) -def detectAndLoad(filename:str? = None, content:bytes? = None) -> Tuple[str,bytes,str[]] = - encoding:str; eol:bytes?; lines:str[] = [] +def detectAndLoad(filename:str? = None, content:bytes? = None, ignoreWhitespace:bool = False) -> Tuple[str,bytes,str[]] = + lines:str[] = [] if filename is not None: with open(encode(filename), "rb") as fd: content = fd.read() - encoding = detectEncoding(content) ?? sys.getdefaultencoding() - eol = eoldet(content) + encoding:str = detectEncoding(content) ?? sys.getdefaultencoding() + eol:bytes? = eoldet(content) if filename is not None: with codecs.open(encode(filename), encoding = encoding) as fd2: lines = safeSplit(fd2.read(), (eol ?? b"\n").decode(encoding)) elif content is not None: lines = safeSplit(content.decode(encoding), (eol ?? b"\n").decode(encoding)) else: return (sys.getdefaultencoding(), b"\n", []) + if ignoreWhitespace: lines[:] = [line.replace("\t", " ").strip() for line in lines] (encoding, eol, lines) -def dataCopy(_tipe:Type[DataType], _old:DataType, *_args, **_kwargs) -> DataType: r = _old._asdict(); r.update(**_kwargs); return makedata(_tipe, *(list(_args) + [r[field] for field in _old._fields])) +def dataCopy(_tipe:Type[DataType], _old:DataType, *_args, byValue:bool = False, **_kwargs) -> DataType = + ''' A better makedata() version. ''' + r:Dict[str,Any] = _old._asdict() + r.update({k: ([e for e in v] if byValue and isinstance(v, (list, tuple, set)) else v) for k, v in _kwargs.items()}) # copy by value if required + makedata(_tipe, *(list(_args) + [r[field] for field in _old._fields])) # TODO also offer copy-by-value here def user_block_input(output:List[str]): sep:str = input("Enter end-of-text marker (default: : "); line:str = sep while True: line = input("> ") if line == sep: break - output.append(line) + output.append(line) # writes to caller-provided list reference def merge( file:bytes? = None, into:bytes? = None, @@ -230,7 +246,8 @@ def merge( mergeOperation:MergeOperation = MergeOperation.BOTH, charMergeOperation:MergeOperation = MergeOperation.BOTH, diffOnly:bool = False, - eol:bool = False + eol:bool = False, + ignoreWhitespace:bool = False ) -> Tuple[Union[bytes,List[MergeBlock]],bytes?] = ''' Merges other binary text contents 'file' (or reads file 'filename') into current text contents 'into' (or reads file 'intoname'), returning merged result. For update, the other version is assumed to be the "new/added" one, while for diff, the current changes are the ones "added". @@ -241,8 +258,8 @@ def merge( ''' encoding:str; othr:str[]; othreol:bytes?; curr:str[]; curreol:bytes? try: # load files line-wise and normalize line endings (keep the one of the current file) TODO document - encoding, othreol, othr = detectAndLoad(filename = filename, content = file) - encoding, curreol, curr = detectAndLoad(filename = intoname, content = into) + encoding, othreol, othr = detectAndLoad(filename = filename, content = file, ignoreWhitespace = ignoreWhitespace) + encoding, curreol, curr = detectAndLoad(filename = intoname, content = into, ignoreWhitespace = ignoreWhitespace) except Exception as E: Exit("Cannot merge '%s' into '%s': %r" % (filename, intoname, E)) if None not in [othreol, curreol] and othreol != curreol: warn("Differing EOL-styles detected during merge. Using current file's style for merged output") output:List[str] = list(difflib.Differ().compare(othr, curr)) # from generator expression @@ -264,9 +281,7 @@ def merge( if len(blocks) >= 2 and blocks[-2].tipe == MergeBlockType.REMOVE: # and len(blocks[-1].lines) == len(blocks[-2].lines): # requires previous block and same number of lines TODO allow multiple intra-line merge for same-length blocks blocks[-2] = MergeBlock(MergeBlockType.REPLACE, blocks[-2].lines, line = no - len(tmp) - 1, replaces = blocks[-1]) # remember replaced stuff with reference to other merge block TODO why -1 necessary? blocks.pop() # remove TOS due to merging two blocks into replace or modify - elif last == "?": # marker for intra-line change comment -> add to block info - ilm:Range = getIntraLineMarkers(tmp[0]) # TODO still true? "? " line includes a trailing \n for some reason - blocks[-1] = dataCopy(MergeBlock, blocks[-1], changes = ilm) +# elif last == "?": pass # marker for intra-line change comment -> add to block info last = line[0] tmp[:] = [line[2:]] # only keep current line for next block # TODO add code to detect block moves here @@ -357,15 +372,10 @@ def lineMerge(othr:str, into:str, mergeOperation:MergeOperation = MergeOperation # TODO ask for insert or remove as well return "".join(out) -def getIntraLineMarkers(line:str) -> Range = - ''' Return (type, [affected indices]) of "? "-line diff markers ("? " prefix must be removed). difflib never returns mixed markers per line. ''' - if "^" in line: return Range(MergeBlockType.REPLACE, [i for i, c in enumerate(line) if c == "^"]) # TODO wrong, needs removal anyway - if "+" in line: return Range(MergeBlockType.INSERT, [i for i, c in enumerate(line) if c == "+"]) - if "-" in line: return Range(MergeBlockType.REMOVE, [i for i, c in enumerate(line) if c == "-"]) - Range(MergeBlockType.KEEP, []) - def findSosVcsBase() -> Tuple[str?,str?,str?] = - ''' Attempts to find sos and legacy VCS base folders. ''' + ''' Attempts to find sos and legacy VCS base folders. + Returns (SOS-repo root, VCS-repo root, VCS command) + ''' debug("Detecting root folders...") path:str = os.getcwd() # start in current folder, check parent until found or stopped vcs:Tuple[str?,str?] = (None, None) @@ -471,7 +481,7 @@ def reorderRenameActions(actions:Tuple[str,str][], exitOnConflict:bool = True) - if exitOnConflict: for i in range(1, len(actions)): if sources[i] in targets[:i]: Exit("There is no order of renaming actions that avoids copying over not-yet renamed files: '%s' is contained in matching source filenames" % (targets[i])) - list(zip(sources, targets)) + list(zip(sources, targets)) # convert to list to avoid generators def relativize(root:str, filepath:str) -> Tuple[str,str] = ''' Determine OS-independent relative folder path, and relative pattern path. '''