Permalink
Switch branches/tags
Commits on Dec 6, 2018
  1. Add IPv6 support to TrustProxy and RobotIP; normalize remote_addr

    machack666 committed Dec 5, 2018
    In order to handle the masking of IPv6 addresses properly, we must first normalize the address to a
    full, expanded form.  This ends up changing (potentially and probably) how the internal
    representation of $CGI::remote_addr is handled, due to expanding the address to its maximal
    representation, with all leading 0's included and no :: truncation present.
    
    This may have side effects to users who are relying on the logging of the IP address in various
    tables, as [data session ohost] will return a string which is 39 characters which may exceed some
    previous limitations in default character size.  (In the Strap catalog, this includes the
    pay_certs_lock table and there are likely many other custom logging solutions for IP which might
    need to be expanded.)
    
    Additionally, because this normalization is done prior to session initialization, it is likely that
    any existing IPv6 sessions could drop due to a different internal MV_SESSION_ID representation.
    Rollout of this should be coordinated to be at a convenient time, or one could write a script to
    process/link all session files which could be affected to include the IPv6 as a hard link to the new
    session file.
    
    The CIDR representation for IPv6 does not need to be in the normalized form, as it will expand
    automatically.  You will also be able to intermix both IPv4 and IPv6 entries for each directive type.
Commits on Dec 4, 2018
  1. Update ip_addr length to support IPv6

    machack666 committed Dec 4, 2018
  2. Sync manifest

    jonjensen committed Dec 4, 2018
  3. Tidy up comments

    jonjensen committed Nov 27, 2018
  4. Remove deprecation comment that won't ever be followed up on

    jonjensen committed Nov 27, 2018
    and clean up some whitespace.
Commits on Dec 3, 2018
  1. Add CIDR support to TrustProxy and RobotIP directives

    machack666 committed Dec 1, 2018
    This works by adding new Vend::CIDR module to turn CIDR formats 11.22.33.44/5 into a regular
    expression corresponding to the valid IPs in this range.  Due to the CIDRs being checked against
    real, valid IPs, we are intentionally not making these patterns as complicated as they might be (for
    instance, we are just checking for any wildcard digit to be /\d{1,3}/ rather than the much more
    complicated regex `/2[0-4]\d|25[0-5]|1\d\d|\d|\d\d/` to explicitly validate the range of bytes
    here.  Because we are comparing against structurally valid IPs, we consider the simplified form to
    be much appreciated.
    
    We also add new parse routine to detect CIDRs for the indicated directives and only apply this logic
    in those cases.  Because we only perform special mapping on the individual fields which already
    match the CIDR format, this should remain backward-compatible with any user-specified regexes in these
    directives in the unfortunate case that they're using shell-style wildcards instead of regular
    regexes.
    
    Vend::CIDR code adapted from the original Javascript at: http://mog.xenowire.net/index.php?cidr2regex
Commits on Nov 26, 2018
  1. Add MYSQL_ENABLE_UTF8MB4 database option

    jonjensen committed Nov 26, 2018
    This sets the DBD::mysql dbh option mysql_enable_utf8mb4.
Commits on Nov 22, 2018
Commits on Nov 13, 2018
  1. Quell error message for harmless empty JSON request body

    jonjensen committed Nov 13, 2018
Commits on Sep 19, 2018
  1. Note that Mark added Braintree payment module

    jonjensen committed Sep 19, 2018
Commits on Sep 18, 2018
  1. Raise minimum Perl version to 5.14.1

    jonjensen committed Sep 18, 2018
    and fill out WHATSNEW document with recent changes.
  2. Fix loop interpolation bug on Perl 5.28

    jonjensen committed Sep 18, 2018
    Without this fix, the Strap demo [if-item-field] ... [/if-item-field]
    loop tags are broken on the flypage and elsewhere.
    
    This bug shows that Perl 5.28 no longer processes string concatenation
    from left to right such that it is safe to autoincrement a variable that
    is referred to elsewhere in the concatenation as well.
    
    I have not seen any other reports of people running into this problem, but
    the new behavior can be inferred from the "perl5280delta" documentation
    which notes under "Performance Enhancements":
    
        Many string concatenation expressions are now considerably faster,
        due to the introduction internally of a "multiconcat" opcode which
        combines multiple concatenations, and optionally a "=" or ".=", into
        a single action.
    
    The perlop documentation for "Auto-increment and Auto-decrement" reads:
    
        "++" and "--" work as in C. [...]
    
        Note that just as in C, Perl doesn't define when the variable is
        incremented or decremented. You just know it will be done sometime
        before or after the value is returned. This also means that modifying a
        variable twice in the same statement will lead to undefined behavior.
        Avoid statements like:
    
            $i = $i ++;
            print ++ $i + $i ++;
    
        Perl will not guarantee what the result of the above statements is.
    
    That does not address changing the variable once and referring to it in
    other places, but now it should.
  3. Fix nonstandard SQL

    jonjensen committed Sep 18, 2018
  4. Quell new warning in Perl 5.28

    jonjensen committed Sep 18, 2018
    Unescaped left brace in regex is deprecated here (and will be fatal in Perl 5.32)
  5. Fix race conditions in timed-build output

    jonjensen committed Sep 18, 2018
    Add a writefile_atomic() routine that is only used for writing complete
    files, not appending or piping, and is atomic on a POSIX filesystem,
    and have timed-build use that.
    
    Reorder the locking in the classic writefile() to return the race
    condition vulnerability window back to the much smaller time it was before
    the introduction of UTF-8 support, to make it very unlikely to occur.
    
    Detailed explanation of the problem and fix:
    
    Recently I was surprised to run into a bug in Interchange's timed-build
    generation routine on a client's production system.
    
    First see the source of &Vend::Interpolate::timed_build and then look
    at &Vend::File::writefile, especially this:
    
        open(MVLOGDATA, $file) or die "open\n";
        if ($encoding) {
            local $PerlIO::encoding::fallback = $fallback;
            binmode(MVLOGDATA, ":encoding($encoding)");
        }
    
        lockfile(\*MVLOGDATA, 1, 1) or die "lock\n";
        seek(MVLOGDATA, 0, 2) or die "seek\n";
        if(ref $data) {
            print(MVLOGDATA $$data) or die "write to\n";
        }
        else {
            print(MVLOGDATA $data) or die "write to\n";
        }
        unlockfile(\*MVLOGDATA) or die "unlock\n";
    
    Note that $file will contain ">/path/to/timed_build_file" so it causes
    the file to be opened in write mode and truncated.
    
    That alone is a race condition: When a file is truncated, there's a
    brief time before it's written to when it's empty and any other process
    reading it will get an empty file.
    
    The introduction of UTF-8 support has made that race window longer because
    right after truncation the binmode() call is made with :encoding(UTF-8)
    or whatever, which can induce dynamic Encode package loads, which is
    relatively slow.
    
    That is one bug, but the bug I encountered is different and much weirder:
    Two timed build files generated by Interchange each had their contents
    doubled.
    
    That is, if the file should have contained "HORSES\n", it instead
    contained "HORSES\nHORSES\n".
    
    I've never seen such a thing before, but it happened in two separate
    timed-build files that both get used right after each other in ITL,
    so if it's caused by a race condition, it's believable that both would
    happen in the same run because they're sequential, so the same thing
    that caused it one place could cause it another.
    
    Now let's walk through the above code as run by two concurrent
    processes to see the race as it happens.
    
    Process A runs this:
    
        open(MVLOGDATA, $file) or die "open\n";
    
    And then process B runs it:
    
        open(MVLOGDATA, $file) or die "open\n";
    
    Process A runs this:
    
        if ($encoding) {
            local $PerlIO::encoding::fallback = $fallback;
            binmode(MVLOGDATA, ":encoding($encoding)");
        }
    
    Then process B runs it:
    
        if ($encoding) {
            local $PerlIO::encoding::fallback = $fallback;
            binmode(MVLOGDATA, ":encoding($encoding)");
        }
    
    Process A runs this:
    
        lockfile(\*MVLOGDATA, 1, 1) or die "lock\n";
    
    Then process B runs it and blocks on the lock:
    
        lockfile(\*MVLOGDATA, 1, 1) or die "lock\n";
    
    Process A runs this:
    
        seek(MVLOGDATA, 0, 2) or die "seek\n";
        if(ref $data) {
            print(MVLOGDATA $$data) or die "write to\n";
        }
        else {
            print(MVLOGDATA $data) or die "write to\n";
        }
        unlockfile(\*MVLOGDATA) or die "unlock\n";
    
    and the timed-build file is written correctly and unlocked.
    
    That unblocks process B which now runs this:
    
        seek(MVLOGDATA, 0, 2) or die "seek\n";
    
    which, oddly for this situation where we are writing to a truncated file,
    seeks to the end of the file! That appears to be a recipe gotten from
    the Perl flock() documentation:
    
        flock($fh, LOCK_EX) or die "Cannot lock mailbox - $!\n";
        # and, in case we're running on a very old UNIX
        # variant without the modern O_APPEND semantics...
        seek($fh, 0, SEEK_END) or die "Cannot seek - $!\n";
    
    and would be harmless if the file is still truncated, but now that the
    file has been written to, means we seek to the end of the file and start
    writing there!
    
    Now process B runs this:
    
        if(ref $data) {
            print(MVLOGDATA $$data) or die "write to\n";
        }
        else {
            print(MVLOGDATA $data) or die "write to\n";
        }
        unlockfile(\*MVLOGDATA) or die "unlock\n";
    
    And we have two copies of the same timed-build output, one after the
    other, in the same file.
    
    This bug has always existed in this code, but I think it became many
    times more likely to be encountered once the binmode() with encoding
    was added because that can be so slow.
    
    There are a few ways to try to solve the doubling problem:
    
    1. Just remove the seek() call which appears to have been only to cope
    with Unix systems that were already old and rare at the time the Perl
    flock() docs were written. Then when the doubling problem hits in the
    future, it'll write the same timed-build output once, then over itself
    again.
    
    If the timed-build runs return output of differing length, it could
    cause the tail end of the longer one to appear after the shorter one,
    which might be an even more confusing bug than this one.  But we could
    call truncate() to empty the rest of the file and remove the risk of
    the longer pieces remaining from before.
    
    This isn't attractive, though, because I don't know for sure that removing
    the seek() is actually safe, and on which systems.
    
    2. Call the binmode() to set the encoding *after* taking the lock. This
    seems like the right thing to do anyway, but it only narrows the window
    for the race condition, and doesn't remove it entirely.
    
    3. Overhaul the routine to use standard practice in writing a shared
    file atomically.
    
    It seems only (3) really solves both of the bugs. We can achieve our
    goal with a procedure like this:
    
    a. Open a new randomly-named temporary file in the same directory.
    
    b. Write to it.
    
    c. Close it.
    
    d. Rename it over the original file.
    
    Renaming is an atomic operation when done within a single POSIX
    filesystem. It isn't atomic for reads over NFS, but writefile() didn't
    support fcntl() locking for NFS anyway, so we're not making that problem
    any worse.
    
    This doesn't stop concurrent timed-build writes from happening, but they
    would harmlessly replace each other in a way that would never leave an
    empty or partially-written file, would keep concurrent reads from ever
    getting a stale timed-build file, and doesn't need locking at all.
    
    Thanks to Mark Johnson for review.
Commits on Aug 13, 2018
  1. Teach send_mail() about MV_EMAIL_CHARSET

    machack666 committed Aug 13, 2018
Commits on Jul 9, 2018
  1. Make Interchange reopen debug.log/STDERR when receiving HUP signal

    machack666 and jonjensen committed Jul 9, 2018
    Co-authored-by: David Christensen <david@endpoint.com>
    Co-authored-by: Jon Jensen <jon@endpoint.com>
Commits on Jun 21, 2018
  1. Fix timecard_stamp() to support a passed timestamp.

    pajamian committed Jun 21, 2018
    timecard_stamp() was originally designed and documented to support the ability
    to pass a timestamp as a second arg, but due to a typo it never has.  This
    commit fixes that issue.
Commits on May 15, 2018
  1. Convert README to Markdown for nice GitHub viewing

    jonjensen committed May 15, 2018
    As suggested in GitHub issue #99.
    
    Along the way, update some things that needed it.
  2. Bump Vend::Util version

    jonjensen committed May 15, 2018
  3. Support recipient addresses in CC and BCC when using Net::SMTP

    pullingshots authored and jonjensen committed May 9, 2018
Commits on May 1, 2018
  1. Remove non-header from BounceReferrals HTTP response headers

    jonjensen committed May 1, 2018
    This invalid extra quasi-header has been there from the beginning of
    this feature, but recent versions of Apache surfaced it when they started
    throwing a 500 error when seeing a non-header in the header data.
Commits on Apr 7, 2018
  1. Fix broken random password number range; consistently make random ins…

    jonjensen committed Apr 7, 2018
    …tead of using zip or phone
    
    It's gross to have this logic duplicated, but fixing that is a task for
    another day.
  2. Newer MySQL versions need ISO timestamp, just like Postgres does

    jonjensen committed Apr 7, 2018
    Tested with Oracle's distribution of MySQL 5.7.
Commits on Apr 6, 2018
Commits on Apr 3, 2018
  1. Fix bug in &Vend::Data::update_data

    jonjensen committed Apr 3, 2018
    I inadvertently introduced this bug in commit fa63936 while optimizing
    a loop. The arrayref passed to set_slice is, in some cases, apparently
    mutated somewhere down the line, though it worked fine in my tests.
    
    Found by Gert van der Spoel <gert@3edge.com>. Thanks, Gert!
Commits on Mar 22, 2018
  1. Remove debugging message from metadata help

    jonjensen committed Mar 22, 2018
  2. add better error handling/messaging when no access to database in Safe

    pullingshots authored and jonjensen committed Mar 13, 2018
    When a db/table has not been pre-opened before attempting to access it in the Safe container, you would usually get a message in the error.log "Can't locate object method open_table via package Vend::Table::DBI"
    
    This commit introduces a more relevant error message along with pointers to help fix the error.
    
    See http://www.icdevgroup.org/pipermail/interchange-users/2018-March/055798.html
  3. Merge PR #87 #87

    jonjensen committed Mar 22, 2018
Commits on Mar 4, 2018
  1. Merge pull request #109 from pullingshots/master

    machack666 committed Mar 4, 2018
    FIX: exclude canceled orders from reports
  2. FIX: exclude canceled orders from reports

    pullingshots committed Mar 4, 2018