Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also .

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also .
base repository: quodlibet/mutagen
Choose a Base Repository
quodlibet/mutagen
24Naman/mutagen
B-Rich/mutagen
Borewit/mutagen
Dark-Matter7232/mutagen
Gabrielcarvfer/mutagen
Holzhaus/mutagen
HongliangZang/mutagen
Ikke/mutagen
MJuddBooth/mutagen
Marchen/mutagen
MaxUdovenko/mutagen
MediaKraken-Dependancies/mutagen
ProjectRecommend/mutagen
Recorichardretardo/mutagen
Reuf12/mutagen-1
RichSr40/mutagen
Sophist-UK/picard_mutagen
TatwiraT/mutagen
Teora/mutagen
TomFryers/mutagen
afcarl/mutagen
akerbis/mutagen
alexliew/mutagen
antlarr/mutagen
anubhav-narayan/mutagen
bamboo321/mutagen
brokkr/mutagen
bugatsinho/mutagen
butihinda/mutagen
cesarecorrea94/mutagen
chucklebot/mutagen
citrin/mutagen
csitko/mutagen
cushy007/mutagen
deng47/mutagen
docbobo/mutagen
donhbryan/mutagen
donovan68/mutagen
drdunks/mutagen
drizzt/mutagen
drone1/mutagen
edent/mutagen
ehermand/mutagen
elbeardmorez/mutagen
erdoukki/mutagen
ethanfurman/mutagen
frestr/mutagen
gaybro8777/mutagen
gdelfresno/mutagen
gergesh/mutagen
ghadesi/mutagen
gmyers18/mutagen
goocy/mutagen
halaei/mutagen
happy-forks/mutagen
hubitor/mutagen
iloveican/mutagen
jack9950/mutagen
jbcurtin/mutagen
jinjing007/mutagen
johnnrain/mutagen
jonntd/mutagen
justin66/mutagen
kendricktan/mutagen
ksato9700/mutagen
kuonanhong/mutagen
lavax/mutagen
lazka/mutagen
levone/mutagen
maphew/mutagen
mars-trueplus/mutagen
maybeRainH/mutagen
mike-matera/mutagen
modimanz/mutagen
musistudio/mutagen
mweinelt/mutagen
mzaman07/mutagen
naglis/mutagen
nchammas/mutagen
oddomatik/mutagen
oscarg933/mutagen
otherjon/mutagen
peppy0510/mutagen
phw/mutagen
point-source/mutagen
pranavg189/mutagen
prokosna/mutagen
samuelnilsson/mutagen
serrasqueiro/mutagen
shikasta-net/mutagen
svetlov97/mutagen
timgates42/mutagen
ttimer/mutagen
uguraba/mutagen
vault-the/mutagen
xjbclz/mutagen
zcm/mutagen
zheaton/mutagen
Nothing to show
base: release-1.45.0
head repository: quodlibet/mutagen
Choose a Head Repository
quodlibet/mutagen
24Naman/mutagen
B-Rich/mutagen
Borewit/mutagen
Dark-Matter7232/mutagen
Gabrielcarvfer/mutagen
Holzhaus/mutagen
HongliangZang/mutagen
Ikke/mutagen
MJuddBooth/mutagen
Marchen/mutagen
MaxUdovenko/mutagen
MediaKraken-Dependancies/mutagen
ProjectRecommend/mutagen
Recorichardretardo/mutagen
Reuf12/mutagen-1
RichSr40/mutagen
Sophist-UK/picard_mutagen
TatwiraT/mutagen
Teora/mutagen
TomFryers/mutagen
afcarl/mutagen
akerbis/mutagen
alexliew/mutagen
antlarr/mutagen
anubhav-narayan/mutagen
bamboo321/mutagen
brokkr/mutagen
bugatsinho/mutagen
butihinda/mutagen
cesarecorrea94/mutagen
chucklebot/mutagen
citrin/mutagen
csitko/mutagen
cushy007/mutagen
deng47/mutagen
docbobo/mutagen
donhbryan/mutagen
donovan68/mutagen
drdunks/mutagen
drizzt/mutagen
drone1/mutagen
edent/mutagen
ehermand/mutagen
elbeardmorez/mutagen
erdoukki/mutagen
ethanfurman/mutagen
frestr/mutagen
gaybro8777/mutagen
gdelfresno/mutagen
gergesh/mutagen
ghadesi/mutagen
gmyers18/mutagen
goocy/mutagen
halaei/mutagen
happy-forks/mutagen
hubitor/mutagen
iloveican/mutagen
jack9950/mutagen
jbcurtin/mutagen
jinjing007/mutagen
johnnrain/mutagen
jonntd/mutagen
justin66/mutagen
kendricktan/mutagen
ksato9700/mutagen
kuonanhong/mutagen
lavax/mutagen
lazka/mutagen
levone/mutagen
maphew/mutagen
mars-trueplus/mutagen
maybeRainH/mutagen
mike-matera/mutagen
modimanz/mutagen
musistudio/mutagen
mweinelt/mutagen
mzaman07/mutagen
naglis/mutagen
nchammas/mutagen
oddomatik/mutagen
oscarg933/mutagen
otherjon/mutagen
peppy0510/mutagen
phw/mutagen
point-source/mutagen
pranavg189/mutagen
prokosna/mutagen
samuelnilsson/mutagen
serrasqueiro/mutagen
shikasta-net/mutagen
svetlov97/mutagen
timgates42/mutagen
ttimer/mutagen
uguraba/mutagen
vault-the/mutagen
xjbclz/mutagen
zcm/mutagen
zheaton/mutagen
Nothing to show
compare: release-1.45.1
  • 14 commits
  • 11 files changed
  • 0 comments
  • 1 contributor
Commits on Jul 11, 2020
T what black defaults to
the gha docs hardcoded it in the matrix example..
the hypothesis tests hang
Commits on Jul 12, 2020
Commits on Jul 17, 2020
It has some problems with Windows file shares where it takes 20x the time
of the fallback implementation and has a similar problem with ZFS on Linux,
see #478.

I've done some basic benchmarks with adding extra padding to 30 flac files each around
20-40MB and there is basically no difference.

So just remove mmap.

Fixes #478
Don't use mmap for rewriting files
Commits on Jul 18, 2020
From 64kb to 256kb. This makes things a bit faster on my machine.
Increasing it even more doesn't change much here.

See #478
Increase the buffer for rewriting files a bit
Commits on Jul 31, 2020
Showing with 89 additions and 260 deletions.
  1. +57 −0 .github/workflows/test.yml
  2. +0 −46 .travis.yml
  3. +9 −0 NEWS
  4. +0 −6 README.rst
  5. +0 −44 azure-pipelines.yml
  6. +1 −1 docs/user/examples/fileobj-iface.py
  7. +1 −1 mutagen/__init__.py
  8. +11 −89 mutagen/_util.py
  9. +2 −1 setup.cfg
  10. +2 −2 setup.py
  11. +6 −70 tests/test__util.py
@@ -0,0 +1,57 @@
name: test

on: [push, pull_request]

jobs:

test:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: [3.5, 3.6, 3.7, 3.8, pypy3]
exclude:
# hangs
- os: macos-latest
python-version: pypy3
- os: windows-latest
python-version: pypy3
steps:
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install pytest hypothesis flake8 coverage
- name: Run tests
run: |
python -m coverage run --branch setup.py test
python -m coverage xml -i
- name: Run flake8
run: |
python -m flake8
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v1

test-docs:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest]
python-version: [3.5, 3.8]
steps:
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install sphinx sphinx_rtd_theme
- name: Build docs
run: |
python -m sphinx -W -a -E -b html -n docs docs/_build

This file was deleted.

9 NEWS
@@ -1,3 +1,12 @@
.. _release-1.45.1:

1.45.1 - 2020-07-31
-------------------

* Fix flake8 tests when run after calling ``setup.py build`` :bug:`482`
* No longer use mmap when rewriting files. Fixes slow save performance with Windows network shares, ZFS and more :pr:`483` :pr:`484`


.. _release-1.45.0:

1.45.0 - 2020-07-11
@@ -18,11 +18,5 @@ is licensed under the GPL version 2 or later.

For more information visit https://mutagen.readthedocs.org

.. image:: https://travis-ci.org/quodlibet/mutagen.svg?branch=master
:target: https://travis-ci.org/quodlibet/mutagen

.. image:: https://dev.azure.com/quodlibet/mutagen/_apis/build/status/quodlibet.mutagen
:target: https://dev.azure.com/quodlibet/mutagen/_build/latest?definitionId=3

.. image:: https://codecov.io/gh/quodlibet/mutagen/branch/master/graph/badge.svg
:target: https://codecov.io/gh/quodlibet/mutagen

This file was deleted.

@@ -101,7 +101,7 @@ def fileno(self):
"""Returns the file descriptor (int) or raises IOError
if there is none.
Will be used for mmap if available.
Will be used for low level operations if available.
"""

raise NotImplementedError
@@ -23,7 +23,7 @@
from mutagen._file import FileType, StreamInfo, File
from mutagen._tags import Tags, Metadata, PaddingInfo

version = (1, 45, 0)
version = (1, 45, 1)
"""Version tuple."""

version_string = ".".join(map(str, version))
@@ -19,19 +19,15 @@
import decimal
from io import BytesIO

try:
import mmap
except ImportError:
# Google App Engine has no mmap:
# https://github.com/quodlibet/mutagen/issues/286
mmap = None # type: ignore

from collections import namedtuple
from contextlib import contextmanager
from functools import wraps
from fnmatch import fnmatchcase


_DEFAULT_BUFFER_SIZE = 2 ** 18


def endswith(text, end):
# usefull for paths which can be both, str and bytes
if isinstance(text, str):
@@ -687,65 +683,7 @@ def seek_end(fileobj, offset):
fileobj.seek(-offset, 2)


def mmap_move(fileobj, dest, src, count):
"""Mmaps the file object if possible and moves 'count' data
from 'src' to 'dest'. All data has to be inside the file size
(enlarging the file through this function isn't possible)
Will adjust the file offset.
Args:
fileobj (fileobj)
dest (int): The destination offset
src (int): The source offset
count (int) The amount of data to move
Raises:
mmap.error: In case move failed
IOError: In case an operation on the fileobj fails
ValueError: In case invalid parameters were given
"""

assert mmap is not None, "no mmap support"

if dest < 0 or src < 0 or count < 0:
raise ValueError("Invalid parameters")

try:
fileno = fileobj.fileno()
except (AttributeError, IOError):
raise mmap.error(
"File object does not expose/support a file descriptor")

fileobj.seek(0, 2)
filesize = fileobj.tell()
length = max(dest, src) + count

if length > filesize:
raise ValueError("Not in file size boundary")

offset = ((min(dest, src) // mmap.ALLOCATIONGRANULARITY) *
mmap.ALLOCATIONGRANULARITY)
assert dest >= offset
assert src >= offset
assert offset % mmap.ALLOCATIONGRANULARITY == 0

# Windows doesn't handle empty mappings, add a fast path here instead
if count == 0:
return

# fast path
if src == dest:
return

fileobj.flush()
file_map = mmap.mmap(fileno, length - offset, offset=offset)
try:
file_map.move(dest - offset, src - offset, count)
finally:
file_map.close()


def resize_file(fobj, diff, BUFFER_SIZE=2 ** 16):
def resize_file(fobj, diff, BUFFER_SIZE=_DEFAULT_BUFFER_SIZE):
"""Resize a file by `diff`.
New space will be filled with zeros.
@@ -782,7 +720,7 @@ def resize_file(fobj, diff, BUFFER_SIZE=2 ** 16):
raise


def fallback_move(fobj, dest, src, count, BUFFER_SIZE=2 ** 16):
def move_bytes(fobj, dest, src, count, BUFFER_SIZE=_DEFAULT_BUFFER_SIZE):
"""Moves data around using read()/write().
Args:
@@ -825,12 +763,11 @@ def fallback_move(fobj, dest, src, count, BUFFER_SIZE=2 ** 16):
fobj.flush()


def insert_bytes(fobj, size, offset, BUFFER_SIZE=2 ** 16):
def insert_bytes(fobj, size, offset, BUFFER_SIZE=_DEFAULT_BUFFER_SIZE):
"""Insert size bytes of empty space starting at offset.
fobj must be an open file object, open rb+ or
equivalent. Mutagen tries to use mmap to resize the file, but
falls back to a significantly slower method if mmap fails.
equivalent.
Args:
fobj (fileobj)
@@ -851,22 +788,14 @@ def insert_bytes(fobj, size, offset, BUFFER_SIZE=2 ** 16):
raise ValueError

resize_file(fobj, size, BUFFER_SIZE)
move_bytes(fobj, offset + size, offset, movesize, BUFFER_SIZE)

if mmap is not None:
try:
mmap_move(fobj, offset + size, offset, movesize)
except mmap.error:
fallback_move(fobj, offset + size, offset, movesize, BUFFER_SIZE)
else:
fallback_move(fobj, offset + size, offset, movesize, BUFFER_SIZE)


def delete_bytes(fobj, size, offset, BUFFER_SIZE=2 ** 16):
def delete_bytes(fobj, size, offset, BUFFER_SIZE=_DEFAULT_BUFFER_SIZE):
"""Delete size bytes of empty space starting at offset.
fobj must be an open file object, open rb+ or
equivalent. Mutagen tries to use mmap to resize the file, but
falls back to a significantly slower method if mmap fails.
equivalent.
Args:
fobj (fileobj)
@@ -886,14 +815,7 @@ def delete_bytes(fobj, size, offset, BUFFER_SIZE=2 ** 16):
if movesize < 0:
raise ValueError

if mmap is not None:
try:
mmap_move(fobj, offset, offset + size, movesize)
except mmap.error:
fallback_move(fobj, offset, offset + size, movesize, BUFFER_SIZE)
else:
fallback_move(fobj, offset, offset + size, movesize, BUFFER_SIZE)

move_bytes(fobj, offset, offset + size, movesize, BUFFER_SIZE)
resize_file(fobj, -size, BUFFER_SIZE)


@@ -7,7 +7,8 @@ omit=

[flake8]
ignore=E128,W601,E402,E731,W503,E741,E305,E121,E124,W504
exclude=mutagen/_senf
exclude=mutagen/_senf,build,dist
max-line-length=88

[tool:pytest]
markers=
@@ -95,8 +95,8 @@ def _check_manifest(self):
assert process.returncode == 0

tracked_files = out.splitlines()
for ignore in [".travis.yml", ".gitignore", ".codecov.yml",
"azure-pipelines.yml"]:
for ignore in [".gitignore", ".codecov.yml",
".github/workflows/test.yml"]:
tracked_files.remove(ignore)

tracked_files = [

No commit comments for this range