@monkeyiq monkeyiq released this Sep 2, 2018 · 15 commits to master since this release

Assets 2

Release Version 2.3

Release date: 3 September 2018.

Distribution

Source snapshots are attached to this announcement and the git tag filesender-2.3 contains the base that these snapshots were created from.

Installation

Documentation is available at http://docs.filesender.org/v2.0/install/

Upgrade Notes

Version 2.2 breaks compatibility with version 1.x. We recommend a fresh installation to version 2.x of FileSender.

Major changes since 2.2

The CI setup has been updated to use the trusty distribution (#400). As part of that move software versions have moved to selenium 4.1 and php 7.0 (required by recent selenium). The phpunit used in CI has also been updated to a more modern version. A new script (https://github.com/filesender/filesender/tree/master/scripts/testing) has been added to create a parallel installation of filesender for running selenium tests locally in docker containers. It is useful to have a parallel install as the config.php file needs to be writeable and is changed in the process of executing the selenium tests.

CI now runs against both mariadb 10.2 and postgresql 9.2 (#407) rather than just the later as in previous versions. CI has been split into 5 concurrent jobs (#410) testing core and selenium execution on both databases with the 5th job testing database migration (on just postgresql currently). This gives great scope for adding new core tests without impacting the overall time taken to run CI on a new pull request.

An attempt was made to split the selenium tests into two concurrent batches but it was found that the overhead of running 7 parallel jobs in travis made the overall execution time gains less than one might have thought. Resulting in execution times in the range of 14 minutes (2 jobs) to 12 minutes (4 jobs) #410 (comment). The current setup uses 2 jobs though this is noted here in case the number of tests is increased and the split is again desired.

The database script now delays creating foreign keys to the end of execution with support for only creating the keys if they do not exist on both mariadb and postgresql (different code needed on both).

The Azure cloud storage has been updated to use the azure-storage-php library (#396)

Configuration changes

No new settings.

Support and Feedback

See Support and Mailinglists and Feature requests.

A new label "release2.3" has been created to specifically track issues relating to this release. Please attach that label when reporting issues that relate to this beta. Current known issues can be seen at https://github.com/filesender/filesender/labels/release2.3

@monkeyiq monkeyiq released this Aug 19, 2018 · 25 commits to master since this release

Assets 2

Release Version 2.2

Release date: 19 August 2018.

Distribution

Source snapshots are attached to this announcement and the git tag filesender-2.2 contains the base that these snapshots were created from.

Installation

Documentation is available at http://docs.filesender.org/v2.0/install/

Upgrade Notes

Version 2.2 breaks compatibility with version 1.x. We recommend a fresh installation to version 2.x of FileSender.

Major changes since 2.1

A very major change to the database schema was performed. The SAML uid is no longer scattered throughout many tables and is instead moved to the new Authentications table. To relate information in the Transfers, UserPreferences, ClientLogs, and guests tables numeric identifiers are now used and referential integrity is in place. More details can be found on the merged PR #382. It is recommended that system administrators make a backup of the database before migrating to the new FileSender 2.2 database schema using the normal scripts/upgrade/database.php process. The database.php is now transaction protected so should either complete fully or roll back all changes if it fails.

Updates to assist in GDPR have been merged including allowing a user to close their account, deleting their files, guests, and other information in the process #387. Cron jobs now allow data to be retired in more places to avoid retaining information that is held too long and does not respect a site's privacy desires. A new privacy page is added allowing the user to see how long the system is configured to retain various things #381.

Support for database views has been added. Default choices allow inspecting the files and transfers tables together as a join, the userpreferences and authentications tables together as a join, and many additional columns showing how "long ago" some events have occurred. This allows finding entries which are too old by simple "more than 30 days old" type queries.

The way encryption keys are generated has moved to using importKey and then deriveKey inline with the recommendation on developer.mozilla.org linked from the merged pull request #375. This introduces a new "key type" and the database has been expanded to handle that and also allow the decryption of existing files as well as files using the new key type. No migration is needed as new files will use the new key type automatically.

Configuration changes

New settings that relate to GDPR:

  • auditlog_lifetime
  • trackingevents_lifetime
  • translatable_emails_lifetime

These work in a similar way to the other lifetime settings and will cause various database tables to be cleaned up by the cron script as data expires beyond the nominated lifetime.

Support and Feedback

See Support and Mailinglists and Feature requests.

A new label "release2.2" has been created to specifically track issues relating to this release. Please attach that label when reporting issues that relate to this beta. Current known issues can be seen at https://github.com/filesender/filesender/labels/release2.2

@monkeyiq monkeyiq released this Jun 30, 2018 · 46 commits to master since this release

Assets 2

Release Version 2.1

Release date: 30 June 2018.

Distribution

Source snapshots are attached to this announcement and the git tag filesender-2.1 contains the base that these snapshots were created from.

Installation

Documentation is available at http://docs.filesender.org/v2.0/install/

Upgrade Notes

Version 2.1 breaks compatibility with version 1.x. We recommend a fresh installation to version 2.x of FileSender.

Major changes since 2.0

A new mode that is on by default will automatically try to resume an upload up to 10 times if a major problem is encountered during the upload. A new option allowing users to send client side logs to the server in the event of a failure. Improved support for running the selenium tests locally as well as in the cloud. Support for encoding generated passwords in a varian't of ascii85. Better random number generator used to generate passwords.

Major changes in 2.0 relative to the 1.x series

It is highly recommended that installations on 1.x consider upgrading to 2.x. There have been security and stability improvements and these will be released in a timely manner for the 2.x series. Incremental updates including database migration will be supported in 2.x.

A short incomplete list of changes: End to end encryption support which is performed entirely in the browser. TeraSender multiple stream upload. Race conditions found and fixed, some edge cases that could result in bad data for some uploads have been fixed. Secondary indexes in the database are now created and used. Multiple storage options are available allowing files to be stored as whole files, chunks, or uploaded to a cloud provider. Filesystem hashing allows multiple NFS servers to be used to store file data in the backend. Zip64 archive generation now makes valid archives for large files. My transfers page is now paginated (and uses indexes) which reduces load times from tens of minutes to seconds for very frequent users. Guest workflow has been improved to not present confusing options. Many security issues have been addressed. Multiple files can now be sent in a single transfer. No flash. And much more.

Configuration changes

ClientLogs which is sending log information from the browser to the server:

  • clientlogs_stashsize
  • clientlogs_lifetime

Automatic resume of upload after a serious error

  • automatic_resume_number_of_retries
  • automatic_resume_delay_to_resume

Ability to run the test suite locally by setting testsuite_run_locally to true.

Ability to set non default (base64) encoding of generated passwords with encryption_generated_password_encoding. Using ascii85 will give slightly higher entropy per byte.

Support and Feedback

See Support and Mailinglists and Feature requests.

A new label "release2.0" has been created to specifically track issues relating to this release. Please attach that label when reporting issues that relate to this beta. Current known issues can be seen at https://github.com/filesender/filesender/labels/release2.0

@monkeyiq monkeyiq released this Jun 7, 2018 · 60 commits to master since this release

Assets 2

Release Version 2.0

Release date: 7 June 2018.

Distribution

Source snapshots are attached to this announcement and the git tag filesender-2.0 contains the base that these snapshots were created from.

Installation

Documentation is available at http://docs.filesender.org/v2.0/install/

Upgrade Notes

Version 2.0 breaks compatibility with version 1.x. We recommend a fresh installation to version 2.x of FileSender.

Major changes since 2.0 release candidate 1

An update for a regression in drag and drop when dirtree was disabled. The download page only offers to send an email if this option is not disabled. Various updates to the docs, particularly the installation information.

Major changes in 2.0 relative to the 1.x series

It is highly recommended that installations on 1.x consider upgrading to 2.x. There have been security and stability improvements and these will be released in a timely manner for the 2.x series. Incremental updates including database migration will be supported in 2.x.

A short incomplete list of changes: End to end encryption support which is performed entirely in the browser. TeraSender multiple stream upload. Race conditions found and fixed, some edge cases that could result in bad data for some uploads have been fixed. Secondary indexes in the database are now created and used. Multiple storage options are available allowing files to be stored as whole files, chunks, or uploaded to a cloud provider. Filesystem hashing allows multiple NFS servers to be used to store file data in the backend. Zip64 archive generation now makes valid archives for large files. My transfers page is now paginated (and uses indexes) which reduces load times from tens of minutes to seconds for very frequent users. Guest workflow has been improved to not present confusing options. Many security issues have been addressed. Multiple files can now be sent in a single transfer. No flash. And much more.

Configuration changes

There are no configuration changes since release candidate 1.

Support and Feedback

See Support and Mailinglists and Feature requests.

A new label "release2.0" has been created to specifically track issues relating to this release. Please attach that label when reporting issues that relate to this beta. Current known issues can be seen at https://github.com/filesender/filesender/labels/release2.0

Pre-release

@monkeyiq monkeyiq released this May 21, 2018 · 69 commits to master since this release

Assets 2

Release Version 2.0 release candidate 1

Release date: 21 May 2018.

Distribution

Source snapshots are attached to this announcement and the git tag filesender-2.0-rc1 contains the base that these snapshots were created from.

Installation

Documentation is available at http://docs.filesender.org/v2.0/install/

Upgrade Notes

Version 2.0 breaks compatibility with version 1.x. We recommend a fresh installation to version 2.x of FileSender.

Major changes since 2.0 beta 4

An upload stall bug that effected larger terasender uploads has been addressed. A situation that could cause a bad chunk of data if a stalled upload was resumed for non terasender uploads has been addressed.

Dirtree support was merged (thanks to Geoff Brimhall). This is a new feature which can be enabled by setting disable_directory_upload to false. Note that only some browsers support dirtrees.

Language files are now generated from the information on poeditor.com. Prior to the import the lang.php for each language was unchanged but sorted to assist in seeing the differences during subsequent i18n imports. New scripts were created to export new terms from the code to poeditor.com for translation. Labels on the graphs can now be internationalized.

Logs can now be saved in JSON format (thanks to Michael DSilva)

Various updates to the docs and CI.

Configuration changes

Raw list of interesting config directives changed or added in this release

terasender_worker_max_chunk_retries
testing_terasender_worker_uploadRequestChange_function_name
disable_directory_upload

Setting disable_directory_upload to false will enable dirtree upload in browsers that support dirtree. terasender_worker_max_chunk_retries sets how many times a TeraSender worker attempts to upload each chunk before signalling failure. The testing_terasender_worker_uploadRequestChange_function_name option is mainly for internal testing and allows a chunk failure to be forced by selecting the name of a fixed prefix javascript function to call to mutilate the response from an XMLHttpRequest.

Support and Feedback

See Support and Mailinglists and Feature requests.

A new label "release2.0rc1" has been created to specifically track issues relating to this release. Please attach that label when reporting issues that relate to this beta. Current known issues can be seen at https://github.com/filesender/filesender/labels/release2.0rc1

Pre-release
Pre-release

@monkeyiq monkeyiq released this Mar 13, 2018 · 92 commits to master since this release

Assets 2

Release Version 2.0 beta 4

Release date: 14 March 2018.

Distribution

Source snapshots are attached to this announcement and the git tag filesender-2.0-beta4 contains the base that these snapshots were created from.

Installation

Documentation is available at http://docs.filesender.org/v2.0/install/

Upgrade Notes

Version 2.0 breaks compatibility with version 1.x. We recommend a fresh installation to version 2.x of FileSender.

Major changes since 2.0 beta 3

New cloud backends to allow storage of file data in Azure or S3. Explicit support for file shredding. Since good shredding can take time FileSender allows that to be done without the user needing to wait in order to continue to use the web interface. New scripts to import translations from poeditor. Update for resuming a failed upload with TeraSender. A new storage type was added to allow file names to be preserved if desired. Minor documentation updates were done on docs.filesender.org

A new config-passwords.php is passed if it exists along side config.php. This allows passwords and other sensitive information to be kept in a separate file to the main configuration for easier sharing of general configuration but not secret information. This is currently mentioned in https://docs.filesender.org/v2.0/cloud/ but may have it's documentation moved in the future.

Translations updated:

  • en_AU, cs_CZ

The following pull requests have been merged:

  • 196 containing multiple small fixes
  • 254 auth url sanitization
  • 266 SSP update for saml libraries

Configuration changes

Raw list of interesting config directives changed or added in this release

storage_filesystem_file_shred_command
storage_filesystem_shred_path
cloud_azure_connection_string
cloud_s3_region
cloud_s3_version
cloud_s3_endpoint
cloud_s3_use_path_style_endpoint
cloud_s3_key
cloud_s3_secret
sysadmin_setting_testcloud

The azure and s3 directives are described at https://docs.filesender.org/v2.0/cloud/
The shredding is at https://docs.filesender.org/v2.0/shredding/

Support and Feedback

See Support and Mailinglists and Feature requests.

A new label "release2.0beta4" has been created to specifically track issues relating to this release. Please attach that label when reporting issues that relate to this beta. Current known issues can be seen at https://github.com/filesender/filesender/labels/release2.0beta4

Pre-release

@monkeyiq monkeyiq released this Feb 10, 2018 · 117 commits to master since this release

Assets 2

Release Version 2.0 beta 3

Release date: 10 February 2018.

Distribution

Source snapshots are attached to this announcement and the git tag filesender-2.0-beta3 contains the base that these snapshots were created from.

Installation

Documentation is available at http://docs.filesender.org/v2.0/install/

Upgrade Notes

Version 2.0 breaks compatibility with version 1.x. We recommend a fresh installation to version 2.x of FileSender.

Major changes since 2.0 beta 2

Zip64 support has been updated. In the previous beta, on the download page if you made a zip that was more than 4gb in size some tools did not want to open the resulting zip file. Zip files should now have no size limit and be more acceptable to more tools and are always zip64 format files regardless of size.

A major new change to help expose issues with upload stalling, change of network or intermittent network absence, laptop sleep and resume, is the display of per worker (Terasender) or single worker (non terasender xhr upload) of staleness information. This is a lead on from the above pull request with pull request 211 which adds the option upload_display_per_file_stats to display the number of seconds each worker has been active on it's current chunk next to the file that the chunk belongs to. While sites may not wish to display this information it is useful to obtain it in order to use it for working out if something has stalled.

FileSender can now use xhr progress messages to keep a much closer eye on uploads for Terasender and single thread file uploads. This allows a stall of a chunk to be known much sooner, for example, if absolutely no bytes are sent by a worker for 5 or 10 seconds then it is likely to be a good bet that something is wrong or stalled. At the moment no active action is taken but PR 221 is a great base for future work restarting stalled chunks.

The upload stall detection code was making assumptions about Workers which might not always be correct. In articular, comparing one worker to another has no guarantee that Worker threads undergo completely fair scheduling. This lead to pr 209.

Two race conditions in upload completion which related to
issue/189 have an update. The issue is that "Sometimes Transfer->makeAvailable() is
called before all Event#file_uploaded are called." A race in both the Terasender and single threaded xhr uploading code lead to pr 206 and 208 respectively.

Better support for Internet Explorer with an issue relating to localStorage being denied (pull 198).

Updates relating to filesystem_hashing have been made to resolve issues/176, 177, and 178.

A fix in pull/214 addresses a potential loop that was in class loading and seemed to only hit some configurations.

More configuration testing is added thanks to issue/197. Issues like 197 are useful because catching and halting on a bad or misleading configuration is likely a better route than undefined or unexpected behaviour through misconfiguration.

Translations updated:

  • French

The following pull requests have been merged:

  • 193 fixes undefined value in Exception::class
  • 199 Fix spelling of 'additional'
  • 37 Show more prominent login button on landing page
  • 210 adds StorageFilesystemExternal to allow storage backed by scripts.
  • 215 builds on 210 to share run() and update the stream library usage a little.
  • 219 displays messages to this effect on attempted download and a banner on the download page to this effect only if crypto is not available and the transfer contains an encrypted file.
  • 204 fixes issues/88 to allow certain cronjobs to run.
  • 218 adds a new configuration option to address issue/80 which requests that crypto transfers do not offer
    a resume transfer option.
  • 244 allows generated passwords to be made longer than the minimum password length
  • 238 pauses an upload if the stop button is pressed
  • 237 normalizes the font sizes on the download page
  • 236 allows the minimal password length to be set
  • 235 allows a maximum file size to be set for encrypted uploads
  • 234 brings back the ability to force the AuP to be selected in order to use the system
  • 37 Show more prominent login button on landing page

Configuration changes

Raw list of interesting config directives changed or added in this release

storage_type
max_transfer_file_size
max_transfer_encrypted_file_size
encryption_min_password_length
encryption_generated_password_length
upload_display_per_file_stats
upload_force_transfer_resume_forget_if_encrypted
upload_considered_too_slow_if_no_progress_for_seconds

There is a new storage_type to allow storage to be handled by scripts.
File sizes can now be limited through the two file size options.
encryption passwords can optionally have a minimal length and an optional larger one for generated passwords.
Encrypted file resume can be forced to be off.
Turning on upload_display_per_file_stats will allow more insight into the health of each chunk being uploaded.

As always, see the main documentation for full details at
http://docs.filesender.org/v2.0/admin/configuration/

Support and Feedback

See Support and Mailinglists and Feature requests.

A new label "release2.0beta3" has been created to specifically track issues relating to this release. Please attach that label when reporting issues that relate to this beta. Current known issues can be seen at https://github.com/filesender/filesender/labels/release2.0beta3

Pre-release
Pre-release

@monkeyiq monkeyiq released this Nov 6, 2017 · 155 commits to master since this release

Assets 2

Release Version 2.0 beta 2

Release date: 6 November 2017.

Distribution

Source snapshots are attached to this announcement and the git tag filesender-2.0-beta2 contains the base that these snapshots were created from.

Installation

Documentation is available at http://docs.filesender.org/v2.0/install/

Upgrade Notes

Version 2.0 breaks compatibility with version 1.x. We recommend to There are currently no detailed upgrade
notes for version 2.0 available.

Major changes since 2.0 beta 1

My Transfers page is now paged. This will improve user experience as the My Transfers page is shown after each upload and could take significant time to load on larger installations.

A meduim sized synthetic database is now shipped with FileSender. These can be created locally but will require over an hour or more depending on how your database is configured. The dataset is now used in the CI set to try to assure that the database can by migrated to the new schema as changes occur.

A new storage type has been merged. This allows files to be stored in chunks instead of as a single block. As chunks are the same size except for the final chunk this might give less filesystem fragmentation over time as files are uploaded and reclaimed. The chunked storage can also be used to hand data to distributed storage systems through FUSE as one installation is doing.

The upload page now has a graph to show upload speeds for the last 30 days, and there are some new stats in the admin page.

Some of the code that generates links will respect the php separator setting allowing you to use ";" instead of "&" so that links are not tempered with by email clients etc.

The REST client now works again.

An upload complete signal can now be issued using the key security. This allows very long running uploads to complete without needing reauthentication in order to signal to the server that the upload is complete.

Locking is no longer performed when writing data. Valid clients should not be uploading the same chunk of the same file at the same time, so file locking does not add any extra validity. This improves performance when storing the /files directory over NFS.

The id column in the StatLogs and AuditLogs are now 8 byte integers. Some installations were bumping into the 3 byte ints used in MariaDB and having issues. The database migration script will move your database tables to this new format:

php /opt/filesender/scripts/upgrade/database.php

A fix for TeraSender and encrypted files #103.

Some of the exceptions in slight SAML configuration mismatches now generate better error messages to try to help you work out what is needed to get the system up and running. It is difficult for new users who have these slight issues to sometimes work out exactly what to change, so any hints the system can give are great for everybody.

Some FileSender exceptions can now optionally log more verbose messages. This will allow system admins who are so inclined to give better feedback to developers who can not reproduce an issue out of production.

FileSender 2.0 beta 1 to 2.0 beta 2 config changes

Raw list of interesting config directives for 2.0 beta 2

storage_type
storage_filesystem_ignore_disk_full_check
storage_filesystem_tree_deletion_command
reports_show_ip_addr
upload_graph_bulk_display
upload_graph_bulk_min_file_size_to_consider

Digging into this a little, storage_type now allows the new storage
of 'filesystemChunked' to be used.

If you are using filesystemChunked then maybe
storage_filesystem_ignore_disk_full_check and
storage_filesystem_tree_deletion_command are of interest. The former can be
useful if you have using FUSE to hand data off to a distributed storage network.

If you would like to not show IP addresses in some reports then set
reports_show_ip_addr to false.

You can turn off the upload speed graph by setting the new upload_graph_bulk_display
to false. It is on by default. The min file size to consider for the graph can be
set using upload_graph_bulk_min_file_size_to_consider which is 1gb by default.

As always, see the main documentation for full details at
http://docs.filesender.org/v2.0/admin/configuration/

Support and Feedback

See Support and Mailinglists and Feature requests.

A new label "release2.0beta2" has been created to specifically track issues relating to this release. Please attach that label when reporting issues that relate to this beta. Current known issues can be seen at https://github.com/filesender/filesender/labels/release2.0beta2

Pre-release

@monkeyiq monkeyiq released this Jul 19, 2017 · 198 commits to master since this release

Assets 2

Release Version 2.0 beta 1

Release date: 19-July-2017.

Distribution

Source snapshots are attached to this announcement and the git branch filesender-2.0-beta1 contains the base that these snapshots were created from.

Installation

Documentation is available at http://docs.filesender.org/v2.0/install/

Upgrade Notes

Version 2.0 breaks compatibility with version 1.x. We recommend to There are currently no detailed upgrade
notes for version 2.0 available.

Security audits

Since the 2.0-alpha1 code has been put through a separate external code security audit:

  • Executed by Radically Open Security

Some issues surfaced and they have where relevant been addressed.

Major changes since 2.0 alpha

Version 2.0 is a new baseline release. Much of the code base was rewritten, a new database design adopted, many configuration directives have been added and existing directives changed.

There is now support for secondary indexes in both database backends. There is also initial movement to greater resilience to 4 byte character encodings and index implementations in MySQL implementations.

Guest implementation has been tested. Specifically how the UI presents itself given various default options in the configuration and situations that are confusing or do not allow the guest to easily progress have been addressed.

The about and help text are now pages instead of dialogs in the web UI. There is also a new provision for sites to present custom content for these pages in specific languages where FileSender updates will not override that content.

Support and Feedback

See Support and Mailinglists and Feature requests.

A new label "release2.0beta1" has been created to specifically track issues relating to this release. Please attach that label when reporting issues that relate to this beta. Current known issues can be seen at https://github.com/filesender/filesender/labels/release2.0beta1

Nov 29, 2016