- nexp
rfc:salotz.001_use-org.org
Proposal: ./rfcs/salotz.001_use-org.org
Abstract:
Use org-mode LOL.
- nexp
rfc:salotz.002_semantic-changelog.org
Proposal: ./rfcs/salotz.002_semantic-changelog.org
Abstract:
A mini-format for git commit messages that supports machine readability (without sacrificing human readability) semantic expression of impacts of code change.
- nexp
rfc:salotz.003_rfc-specs
Proposal: ./rfcs/salotz.003_rfc-specs.org
Abstract:
Specifications for necessary information for an RFC and how to format it.
- nexp
rfc:salotz.004_nexps
Proposal: ./salotz.004_nexps.org
Abstract:
A proposal that defines how to name digital document entities: e.g.
format:namespace.field-1_field-2
- nexp
rfc:salotz.005_syxel-buffers
Proposal: ./rfcs/salotz.005_syxel-buffers.org
Abstract:
Format that utilizes a granular unit of ‘syxel’ (symbolic pixel) in a rectangular plane similar to a pixel based image format. Syxels are basically character points.
- nexp
rfc:salotz.006_codetags
Proposal: ./rfcs/salotz.006_codetags.org
Abstract:
Tags that are added in comments to code that add semantic meaning to otherwise freeform comments, making them searchable by machine and available to tooling.
- nexp
rfc:salotz.007_py-lib-layout
Proposal: ./rfcs/salotz.007_py-lib-layout.org
Executive Summary:
A layout and content specification that follows best practices that strives for effectiveness rather than complete compatibility.
- nexp
rfc:salotz.008_linux-shell-startup
Proposal: ./rfcs/salotz.008_linux-shell-startup.org
Executive Summary:
A collection of script files which add a layer of indirection to standard linux shells (i.e. POSIX
shandbash) that allows for a more modular, composable, and semantic organization.
- nexp
rfc:salotz.009_repo-layout
Proposal: ./rfcs/salotz.009_repo-layout.org
Executive Summary:
A template format for generating and specifying repo layouts in directories, i.e. cookiecutter.
- nexp
rfc:010_growth-versioning
Proposal: ./rfcs/010_growth-versioning.org
Executive Summary:
Version formatting that doesn’t lie.
- nexp
rfc:011_write-tags
Proposal: ./rfcs/011_write-tags.org
Executive Summary:
Tags (see rfc:salotz.006_codetags for a similar proposal) that are added in comments to prose (documentation, whitepapers, technical documents, etc.) that add semantic meaning to otherwise freeform comments, making them searchable by machine and available to tooling.
- nexp
rfc:012_information-errors
Proposal: ./rfcs/012_information-errors.org
Executive Summary:
Errors in information systems, like event logs, should take on a role of providing information rather than being categorized based on their operational importance as is the case in most languages (e.g. checked vs. unchecked exceptions). Inspired by Stuart Holloway and specifically this talk: https://youtu.be/oOON–g1PyU?t=965
- nexp
rfc:013_code-targets
Proposal: ./rfcs/013_code-targets.org
Executive Summary:
Code targets are a specification for a syntax which should be recognized by metaprogramming tools like template engines, editors, etc.
- nexp
rfc:salotz.014_project-intent
Proposal: ./rfcs/salotz.014_project-intent.org
Executive Summary:
A proposal for a best practice that includes a clear statement of intent on open source projects that would allow for greater understanding by consumers about the current state and intentions of developers in fairly unambiguous terms. Extensions could provide optional semantic vocabularies for very clear intent and to provide a set of choices for developers so they don’t have to think as much from scratch.
- nexp
rfc:salotz.015_collab-line-editing
Proposal: ./rfcs/salotz.015_collab-line-editing.org
Executive Summary:
A protocol and format to facilitate low-friction editing of line-oriented document formats like LaTeX, Markdown, Org-mode, etc.
- nexp
rfc:salotz.016_trivial-plaintext-formats
Proposal: ./rfcs/salotz.016_trivial-plaintext-formats.org
Executive Summary:
A small collection of nearly trivial plaintext formats along with file extensions. Includes things like a line based list format.
- nexp
rfc:salotz.017_bunker
Proposal: ./rfcs/salotz.017_bunker.org
Introduces the concept of a
bunkerdirectory for user only data in their respective $HOME directories which is simply.${USER}or.${USER}.d. This is to allow a safe-space for customization and configuration that will be unmolested by other programs since it is unlikely that they will have this hardcoded in their behavior.
- nexp
rfc:salotz.018_secret-database
Proposal: ./rfcs/salotz.018_secret-database.org
A specification for a secrets database using GPG and mostly compatible with tools like
pass.
- nexp
rfc:salotz.019_MPP
Proposal: ./rfcs/salotz.019_MPP.org
Protocol for live-updating of file-based development projects.
- nexp
rfc:salotz.020_repo-issue-tracker
Proposal: ./rfcs/salotz.020_repo-issue-tracker.org
Executive Summary:
Schema for including issue tracking sources with a project, without having to rely on outside forges.
- nexp
rfc:salotz.021-org-mode-ng
Proposal: ./rfcs/salotz.021_org-mode-ng.org
Executive Summary:
Proposals for the next generation of org-mode involving simplifying, modularizing, and making it more extensible and amenable to a wider non-emacs ecosystem.
- nexp
rfc:salotz.022_template-inheritance
Proposal: ./rfcs/salotz.022_template-inheritance.org
Executive Summary:
Taking inspiration from gitignore.io and their template structure. Specifies a way to layout a repository of templates that allows for: stacks, inheritance, and ordering.
- nexp
rfc:salotz.023_editor-rfis
Proposal: ./rfcs/salotz.023_editor-rfis.org
Executive Summary:
Publish a series of RFI standards for editors that describe the behavior of specific features.
- nexp
rfc:nextgen-vector-graphics-formats
Proposal: ./rfcs/nextgen-vector-graphics-formats.org
Executive Summary:
Specifications for a suite of vector graphics formats, namely providing both plain-text and efficient binary representations suited for different kinds of applications.
In addition to merely specifying best practices, protocols, and standards via RFCs it has come to my attention that there is a need for a listing of concrete proposals for actual working software that perform critical functionality.
The purpose here is not to “shout at clouds” but to attempt to actually evaluate and validate the need for the introduction of new software.
In addition to describing the purpose and semantics of the propsed software in-depth analyses of prior art should be done as well with feature matrices showing gaps in the current functionality.
It is unclear about how exactly to incorporate the idea of the ease of integration (or perhaps “civility”) of a piece of software but attempts should be made to take this into account. E.g. is it open-source and licensed in such a way that it could be used in perpetuity.
Name is inspired by the Scheme Request For Implementations (SRFIs).
Categories of reflection capable for data objects:
- “black” objects
- opaque binary blobs
- “grey” objects
- e.g. PostScript files, XML, image formats
- specialized and customizable diffing functionality
- “white” objects
- those with well-known (semi-universal) diffing strategies and merge techniques
Ideas:
There should be a middle-ground between Decentralized and Centralized VCS which is more similar to federated systems.
The idea is that authority over the “master copy” (centralized vs. decentralized) is controlled (by some means) and can be delegated to a collection of hub servers.
A purely decentralized system (like git) is fundamentally unable to handle black objects because the source of these can only be controlled either through shear replacement (called arbitrary replacement in which no merge strategy is possible and decisions are completely arbitrary and determined by authority) or through locking (change control again completely arbitrary if we are to avoid “wars” between users).
While both arbitrary replacement (arbitration) and change control (delegation) are viable solutions in general for dealing with black objects, they can’t be implemented with git without an extension to the protocol. These protocol extensions are typically performed by the hosting service which controls the “master copy” (like github or gitlab, called forges).
This leads to lock-in and/or a lack of ability to cooperate with others and essentially a centralized, non-distributed paradigm that allows for offline work encoded as a temporary “fork” of authority, despite the “decentralized” moniker.
While any new VCS system should surely support offline-work it should be reified in the protocol, rather than being implemented as a temporary fork.
Furthermore, the usage of the two strategies for blackish objects (arbitration and delegation) should also be reified and be able to be composed.
Here is an example:
Lets say you are an organization that is working on a large product and you have more than one team.
You don’t want the progress of one team to hold up the other one and you would like them to work completely in parallel.
However, you don’t want them to make mutually incompatible changes to objects for which they really should be cooperating with other teams on (particularly to blackish objects which can’t be intelligently merged).
So you split up the entire repo image (the “monorepo” perhaps) into several regions:
- team A’s region
- region A
- team B’s region
- region B
- shared resources region
- region S
- innaccessible region (for other teams or administrators only etc.)
- region X
This allows for work done region’s A and B to happen quickly and according to the self-organization of the respective teams.
This forbids access to region X which should either be irrelevant parts of the monorepo or sensitive reflective configuration data that should only be controlled by the reigning authority (such as the access control itself).
Region S however both teams can use and merges therein should be performed by an authority.
However, within this shared region we still want to implement locks for blackish objects or implement replacements.
These should then be determined by the overarching authority (for consensus) and as such must be centralized. Remember consensus means slow.
Because we are able to isolate only the few contentious resources both teams must access regions A and B need not gain consensus between them.
So in effect we have a hybrid decentralization and distribution strategy that is allowed by a hierarchy in which authority is delegated to allow for parallel work to be done, while still maintaining consensus.
The main idea behind my refugue project is that you can manage your own personal network (PN) while using something like the internet or sneakernet to actually perform the transfers etc. Kind of like urbit.
The point being that at a semantic level the personal network is what matters to an individual or enterprise and not really the internet or what ICANN says about domain names.
This is something I always struggled with because it felt that control of my own network was out of my hands. In reality ICANN and the internet simply enable the underlying transport to take place conveniently and to build my network on top of.
This network really isn’t that complicated and simply relies on some mapping of unique pet names to a collection of addresses which you may find that peer on. Which can be:
- IP addresses
- domain names
- zeronet addresses
- file paths to mounted volumes
etc.
The same is useful for a distributed/decentralized version control system. And is useful for allowing polymorphism in the data that resides on each peer.
For example I want to check out just a portion of the enterprise monorepo because I either don’t need all of it, can’t hold all of it, or am not allowed to hold all of it.
Specifying a network from a single holder of authority is a way to achieve this.
How consensus on this single point of authority is decided can be customized but can be something like:
- zookeeper or raft
- blockchain
- held in a person i.e. CEO
- boar
- git-lfs
- git-annex
- subversion
https://github.com/dspinellis/git-issue
Pros:
- import/export to common forges like github and gitlab
- implemented in shell scripts
Cons:
- fixed schema, oriented around small files of undescribed schemas, non-extendable
- only works with git
- not editable directly via editor easily:
- metadata and description spread across many files for each issue
- file/dir names based on hashes rather than anything meaningful
https://github.com/npryce/deft
Pros:
- VCS agnostic
- Text-editor friendly
- issues live in a renamable dir
- only 2 files: description + metadata
- files named after file
Cons:
- Python 2
- no forge integration
- must manually name issues
https://github.com/neithernut/git-dit
After reviewing the existing options I like the UX of deft the best
but it suffers from some implementation issues mainly being
implemented in Python 2.
I want my issues to be able to be edited by a text editor as not onyl a secondary means, but as a first class thing.
This doesn’t preclude command line tools for this purpose but using only a text editor should be ergonomic.
Ideally anyhow it would get integrated to emacs or VSCode etc. which is how these kinds of tools work better anyhow.
Each issue should be only a single file.
It is too annoying to switch between different files to tweak and twiddle metadata.
In the limit individual issues could be of different actual formats that allow for varying degrees of semantic data. (similar to git commits).
Issues can be in SOML (my minor TOML variant for writing lots of text) where you can add tags, flags, and metadata as you please alongside long descriptions in different markup languages.
They can also be in other formats as you wish and should be a decision each project should make.
That is the schema/protocol for the directory layout etc. should be orthogonal to the content of the actual issues.
This can only really be achieved with single file issues.
However, there should be affordances for attaching resources (like images etc.) into sidecar directories. But these really should contain any issue writer writings.
Other options for issue formats could be:
- skribilio
- org-mode
- markdown
- plaintext
- XML, HTML
- JSON
Here is an example with SOML:
First line of the issue
[meta]
status = 'open'
assigned = '@salotz'
tags = [
'feature',
'bug',
'critical',
]
[description]
Here is the longer description of the issue describing what to do
about it.
It should allow introspection with links and such.
Here I am linking the path to a piece of code I want to reference:
[[proj:/src/package/__init__.py?line=30;col=12][code here]]
[discussion]
[[comments]]
contributor = '@salotz'
message = [
I think this is a good issue, I can even do this myself.
]
[[comments]]
contributor = '@wumpus'
message = [
I disagree, I don't think its worth our time.
]
Its not always easy to manually name issues. That is why things like github and gitlab use auto-naming things where they use numbers to refer to them.
Tools should support this so that people don’t have to name their issues.
However, to keep things friendly the file names should support adding extra “fields” to add short descriptions.
Examples:
An uncommented issue should just be a number. Issues should increase monotonically and start at 0 and should include 0 padding for sorting of crummy legacy software. All tools should sort numerically.
The zeroth issue:
000000.issue.schema
The next issue:
000001.issue.schema
The issue after with a name issue-key also:
000002_issue-key.issue.schema
I was looking for at one point an alternative to the standard library.
Here is the example:
the shutil module is only good in 3.8+ because of some basic options
in the copytree function, and so it is pretty unreliable and
confusing when you are trying to build standard tooling that should
use that.
Typically in tooling I would write:
cx.run("cp -r tree ~/scratch/tree")But this is POSIX only, and using the python standard library should be cross platform, right?
So I change to using shutil:
import shutil as sh
sh.copytree('tree', "/home/salotz/scratch/tree")This will fail if “/home/salotz/scratch/tree” already exists. And is
annoying, so you add the exist_ok option but this only is in python
3.8....
A lot of my tooling uses 3.6 or 3.7 because dependencies in other environments. This is super annoying.
Also there is no features in py 3.8 that make shutil able to gain this super power. Its just an if statement.
For something like this I wish there was just another library I could import to get this behavior on all relevant python versions.
import altstd.shutil as shThe problem is that there are loads and loads of utilities that are very useful for doing common patterns or used during development (i.e. during debugging & prototyping etc.), and these are scattered across dozens and dozens of little tiny packages across the internet.
This leads to a lot of issues that are well-known. For example you run into the famous “left pad” problem. Which is where one of these tiny projects becomes unavailable and you lose that dependency and break all your code.
Other problems are that when you get into real projects there is a
certain friction associated not only with making sure that these
little tools are specified in the env files (i.e. requirements.in
etc.) and that every variation of these files gets updated as well.
On top of that you simply have more dependencies that you have to manage and worry about breaking your code when they change something.
My idea is to go around the internet and collect all of these little utilities into a single repository that is tracked as a single dependency.
I would call this the “Non-Standard Standard Library” or the ‘NSSL’.
(Name liable to change)
Some candidates that motivated this:
A collection of custom diffs that can be plugged into a variety of things like VCS.
Add support for things that aren’t amenable to standard GNU diff like things.
The gitignore.io collection is a great resource with a terrible website web API, local server mess used to implement it.
Something much simpler should be devised that simply works with either local repos or from a git repo.
You should also be able to combine different repos together.
For instance I like to use community maintained gitignores for a lot of stuff because then I don’t have to worry about stuff. But I typically end up home-growing my own sets of ignores that I use across many projects and I want to be able to compose them all.
For instance you can make a single gitignore using the query system they have in some command line tools as well:
giig python,emacsWill give you both python and emacs.
One: I want to be able to specify which repo to use for this.
giig --url git+https://github.com/salotz/gitignore-salotz.git python,emacsFor lets say my personal one. But I also want to be able to compose them. (Do we need this to support that or can I just use piping?)
Perhaps above this would implicit compose that and the default community one, to only get mine I might use:
giig --no-default \
--url git+https://github.com/salotz/gitignore-salotz.git python,emacsThis is necessary for if there is conflicting names. I.e. both have
Python.gitignore in them.
To allow for composition you can use in the secondary repos the name
Python.patch or specify some stacks Salotz-Python.Python.stack
which is a symlink to Python.stack and infers that we also want to
use the Salotz-Python.gitignore template file.
And for instance combining the default repo with a local one:
giig --url file://~/my-ignores python,emacsI have come across a number of tools which read your source code and then generate some set of quality metrics or other representations of your code that allows you to do high-level planning and understanding about your code base.
(I am primarily interested in Python although this niche is necessary for really any code base.)
All of these tools have to reinvent the same wheel which is generating an extended “AST” which is over the files in your project.
The touchstone for this is my pymatuning which is a good prototype of this, albeit surely incomplete in many ways.
So the proposals for software would then be:
- reliable general purpose tree data structure in python that allows for standard querying and outputs to various common formats like JSON etc.
- a standard Python module/package representation using this tree data format.
Should implement this in pure-python for maximum portability.
This tool converts from bytes to higher units correctly and with optional formatting help.
After having used Prometheus for some projects for monitoring programs, I find the model usable and a niche I haven’t found in any existing profiling software.
Often profilers (at least in the python world) inject code into your program and get metrics on everything that is going on.
This is useful, but in many scenarios the results are overwhelming and quite complex.
The Prometheus approach is much like a traditional logger in that you have to actually write code to make data samples. Prometheus differs in that it supports structured data (albeit a usefully small number of things like Gauges, Counters, etc.) while logging is just some strings usually and is then queried using a full-text search engine.
The full text search on logs seems, unsatisfying. Not only have you gone the extra mile to both add logging in and collect performance data (object sizes timings etc.) but you now have a separate problem of parsing them.
Having first class support for the most common monitoring use cases is something that should be supported.
Logging software like eliot provides a vastly improved experience on
standard loggers by outputting a JSON stream of log messages. By
leveraging this you can report all kinds of data including numeric
data.
As an aside eliot has a focus on root cause analysis and not
monitoring, and as such it has a much higher overhead in injecting the
code into your “business” code. I.e. it wouldn’t really be amenable in
all cases to “aspect oriented” approaches like prometheus gauges would
be. That said you don’t need to use these features in eliot.
The extra things on top of eliot that a Prometheus like system would
have, is a special purpose time series database and query engine
(PromQL).
The problems with Prometheus are that it is meant to be “cloud
native” and as such is best deployed in a container cluster along with
a bunch of other helper services that communicate via HTTP.
While this is an approach suited to a networked environment it increases the overhead of getting up and started by a lot.
The goal of this project would be to significantly reduce this complexity and not need to have http servers running for all programs being scraped.
This could simply be done over a RESTful virtual file system like 9P on FUSE.
Other than that prometheus places a great importance on shipping as a single binary (as most golang programs do) which is useful for moving it around and easing deployment.
In our situation we could be more oriented towards plugins and shipping more like a traditional python framework.
Plugins then would essentially replace the numerous side-car services that prometheus talks to (like the alertmanager) which are just “dockered” away so that the prometheus binary remains easy to deploy.