reflow0.6.8

Reflow 0.6.8 is a bugfix release containing a fix for configuration validation. It also contains some minor runtime improvements.

reflow0.6.7

This release contains a number of improvements and features.

• reflow collect: tool to perform garbage colection
• eval: support for tracing through AWS xray
• tool: support for bundling and running bundles (snapshots of the set of modules that constitute a run)
• syntax: permit indexing of lists
• syntax: improved error messages
• reflow repair: a repair tool to assist with forward-migration of cache schemes
• eval: print exec resource info in debug logs
• aws: disallow remote credential providers when config aws="awsenv"
• aws: load shared configuration by default (e.g., AWS region)
• syntax: compute digests for range builtins
• eval: fix concurrency bug in continuation resolution
• tool: show keepalive time in list output
• eval: save exec profiles to cache
• docker: upgrade to newest docker client library
• docker: more aggressive image pull retries

Note that this release has a new minimum CoreOS AMI requirement when running on AWS EC2. Reflow will migrate your configuration for you:

$ reflow run simple.rf 
invalid configuration: outdated CoreOS AMI: please run "reflow migrate"
$ reflow migrate
reflow: creating DynamoDB table marius-reflow-meta
reflow: dynamodb table marius-reflow-meta already exists
reflow: dynamodb index ID4-ID-index already exists
reflow: migrated ec2 AMI to ami-4296ec3a
$ reflow run simple.rf 
reflow: run ID: 29915fcc
...

reflow0.6.3

Reflow0.6.3 is a maintenance release with the following changes:

• ec2cluster: config-overridable probe depth
• config: derive regions in awsenv and ec2metadata
• docker: use $DOCKER_HOST if it set (for local or hybrid modes)
• tool: restore pprof handlers

reflow0.6.1

This is a bugfix release that fixes an intermittent error caused by an unsafe system call.

reflow0.6

This release introduces many improvements to Reflow's user experience and performance and, as always, many bug fixes and smaller improvements.

The major changes in this release are: bottom up evaluation, which allows Reflow to use both logical and physical cache keys in order to improve cache utilization and hit rates; lazy object fetching which minimizes the amount of data transfer Reflow needs to perform fromcache; and status displays which, when Reflow is run in a capable terminal, displays current evaluation status to the user, so that it's always simple to understand what Reflow is doing at a glance. Many of Reflow's commands are also clarified and simplified. For example "reflow info" is now capable of displaying information about any valid Reflow identifier (e.g., a cached object, a running exec, an alloc, or a cached fileset).

Reflow 0.6 also introduces some changes in how cache entries are stored. This is in order to support abbreviated lookups of cache entries (e.g., "reflow info abcdef"). Command "reflow migrate" migrates the current database setup to support this feature.

The changes and improvements are detailed below.

Core

• bottom-up evaluation: With "reflow run -eval=bottomup", reflow performs pure, cache-assisted bottom-up evaluation. This allows Reflow to compute physical cache keys in addition to logical ones, and can thus (safely) cache more widely. Bottom-up evaluation logs also never skip any part of the evaluation, thus providing a more sensible program trace. In bottom-up mode, Reflow looks up both keys.
• precise, lazy object fetching: Reflow now fetches objects from cache by need, and not when the corresponding cache entry is retrieved. This means that Reflow now never fetches more data than is absolutely required.
• many improvements to evaluation debug logging (reflow -log debug run)
• add configuration provider for static clusters
• extensible resource types (beyond cpu, mem, disk)
• use pure concurrency limits for transfers
• a number of cluster scheduler tweaks and improvements
• add a status reporting facility that is used to report the current set of evaluation activities (executions, interns, data transfers, cluster management activities) to the user in a comprehensible way
• identifier-based invalidation: 'reflow run -invalidate=regexp ..." will invalidate cache entries produced by nodes whose identifiers match the provided regular expression. For example, the command "reflow run -invalidate=[aA]lign wgs.rf ..." will run wgs.rf as normal, but recompute all alignments.

Tool

• unified handling of sha256s: "reflow info" can now resolve any valid Reflow identifier
• "reflow info" can now resolve sha256 prefixes; this requires adding indices to DynamoDB table with new command "reflow migrate"
• command "reflow cat" writes a cached object to stdout
• command "reflow sync" synchronizes a cached fileset to a local destination
• "reflow doc" displays all available system modules when invoked without arguments
• "reflow doc" now renders all default parameters by pretty-printing their expressions in an abbreviated form
• handle interrupts (control-c) properly, so that allocs can be reclaimed much more quickly after an interrupted run

Syntax

• comprehension support for cross products and filters
• $/regexp.Match: regular expression matching
• $/regexp.Replace: regular expression matching with replacement
• $/strings.FromFloat: convert floating point numbers to strings
• $/strings.Join, $/strings.HasPrefix, $/strings.HasSuffix: string utilities
• support missing arithmetic binary operations for ints and floats: %, <<, >>, -, /
• improve error messages in various standard library funcs
• various improvements to type and syntax error messages
• add registration mechanism for custom intrinsics
• support for expressing cpu features in resource requirements (e.g., "intel_avx", "intel_avx2", and "intel_avx512" as supported by various EC2 instance types)
• trace builtin to aid debugging
• support fractional CPUs in resource requirements
• range builtin: create integer ranges
• support rich(er) key types in maps: string, int, float, bool, and file are now supported

Cache

• record latest access times counts to permit for LRU collection
• support digest abbreviations

Local executor

• don't attempt to hardlink across devices
• more careful about parsing Docker repository names to avoid unnecessary pulls

ec2cluster

• update definitions from Amazon
• change resource fudge factor; instance-reported resource availability changed with recent virtualization changes
• remove idle boundary computation no longer needed due to EC2 billing changes; instead use a straightforward idle timeout
• user-mergeable cloud-config
• instances launched by ec2cluster may now be assigned a (configurable) EC2 instance profile
• batch DescribeInstance calls with paging to reduce EC2 API load
• support for avx, avx2, and avx512 resource constraints
• allocation improvements resulting in better instance packing
• configurabhle striping of multiple EBS volumes to improve throughput; with this change we can push >2GiB/s on larger instance types

reflow0.5.3

This release contains support for AWS's new C5 instance types, and a number of bug fixes and improvements.

• ec2cluster: support for the new c5 instance types
• local/s3: remove digest.WriterAt until we have the new S3 downloader
• syntax: fix miscellaneous delayed evaluation bugs
• tool: check if shell returns an error
• local: use the return code from Docker's ContainerInspect call
• syntax: increase local file limit from 100MB to 200MB
• local: fix invalid sequence token assignment in cloudwatchlogs
• local: restore s3 state across restarts

reflow0.5.2

• reflowlet: terminate on 10-minute idle time (since AWS does secondly billing)
• syntax: fix digest for ExprApply
• syntax: {files,dirs}.Copy: return a helpful error message if URL scheme is missing
• runner: improve stealer allocation
• eval: scramble digest when an OpVal carries an error
• syntax,types: fix type environment for recursive module instantiations
• reflow shell: new command to shell into an exec
• tool: use stdout for reflow doc output
• tool: disable -gc for v1 flows, -dir alias for -localdir, for backwards compatibility
• reflow logs -f: support for tailing ongoing exec logs
• tool: restore noexterncache flag
• syntax: always sort map keys carefully before digesting
• syntax: fix environment persistence for declarations
• doc: add language overview docs
• config: add "local" configuration provider for user
• eval: print full image name in debug logs
• syntax: add few missing cases for trailing commas
• syntax: add a semicolon at the end of a file if there is a missing newline
• syntax: add arbitrary precision float, conversion between int and float
• eval: add option/flag -recomputeempty to recompute empty cache values

reflow 0.5.1

Reflow 0.5 was built with an incorrect Go library vendor configuration, which would cause intermittent Docker errors. This release is exactly as reflow 0.5, but with the correct vendor configuration.

reflow 0.5

This is the first public release of Reflow, enjoy!

Binaries are provided for macOS/amd64 and Linux/amd64; please see the project README for information on how to build binaries for other architectures.