Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fatal: unable to write new index file #3181

Closed
yarikoptic opened this issue Feb 26, 2019 · 5 comments
Closed

fatal: unable to write new index file #3181

yarikoptic opened this issue Feb 26, 2019 · 5 comments
Assignees
@kyleam
Labels
fix-implemented A fix is available, but has not been merged or released, yet. severity-normal standard severity

Comments

@yarikoptic
Copy link
Member

git for some reason refuses to write index under /tmp which has a good amount of space left, and resides on a different partition:

$ datalad create -d . --force Badal
[INFO   ] Creating a new annex repo at /data/.../sourcedata/Badal
[ERROR  ] Cmd('/usr/lib/git-annex.linux/git') failed due to: exit code(128)
|   cmdline: /usr/lib/git-annex.linux/git -c receive.autogc=0 -c gc.auto=0 read-tree --index-output=/tmp/datalad_temp_7drd148k 443e4211f974430771bebc575230fece04085620
|   stderr: 'fatal: unable to write new index file' [cmd.py:execute:821] (GitCommandError)

and indeed so even with system wide git

$ git -c receive.autogc=0 -c gc.auto=0 read-tree --index-output=/tmp/datalad_temp_7drd148k 443e4211f974430771bebc575230fece0408562
fatal: unable to write new index file

and there is enough space there

$ df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
/dev/md7        9.4G  5.7G  3.7G  61% /tmp

@kyleam are you aware of any obvious reasons (cross-filesystem?) which would cause the problem with temporary index? I guess we better use .git/tmp for it
@kyleam
Copy link
Contributor

kyleam commented Feb 26, 2019

I didn't know that, but yeah, guess it needs to be created on the same file system.

I guess we better use .git/tmp for it

Sounds good, though I'd prefer a more specific name: ".git/datalad-temp.index" or even a temporary file name with a "datalad-" prefix.

@kyleam kyleam added the severity-normal standard severity label Feb 26, 2019
@kyleam kyleam self-assigned this Feb 26, 2019
@kyleam
Copy link
Contributor

kyleam commented Feb 26, 2019

Ah, from the lovely manual:

       --index-output=<file>
           Instead of writing the results out to $GIT_INDEX_FILE, write the resulting index in the named file.
           While the command is operating, the original index file is locked with the same mechanism as usual. The
           file must allow to be rename(2)ed into from a temporary file that is created next to the usual index
           file; typically this means it needs to be on the same filesystem as the index file itself, and you need
           write permission to the directories the index file and index output file are located in.

kyleam added a commit to kyleam/datalad that referenced this issue Feb 26, 2019
@kyleam
BF: annexrepo: Create temporary index under .git/
8ffe00d 
Write the temporary index file under the git directory to ensure that
the it resides on the same file system as the main index file.  See
the description of `--index-output` in `git-read-tree`s manpage.

Fixes datalad#3181.
@kyleam kyleam mentioned this issue Feb 26, 2019
Merged
@kyleam kyleam added the fix-implemented A fix is available, but has not been merged or released, yet. label Feb 26, 2019
@kyleam
Copy link
Contributor

kyleam commented Feb 27, 2019

@yarikoptic What's your preference on closing via the 0.11.x branch? Should we leave issues that were fixed in 0.11.x open until a periodic merge of 0.11.x into master or would you prefer to close manually? I suppose one advantage of the former is that it serves as a reminder to do the merge.

@yarikoptic
Copy link
Member Author

I think it makes sense to have them indeed closed by merge into master, avoiding manual

@vassiki
Copy link

vassiki commented Mar 2, 2019

Happy to report that I didn't run into this bug while using the branch 0.11.x as a developer. Version of datalad was 0.11.3.dev13. Thanks @yarikoptic and @kyleam!

@kyleam kyleam closed this as completed in 24ffcbb Mar 2, 2019
yarikoptic added a commit that referenced this issue Apr 6, 2019
@yarikoptic
Merge tag '0.11.4' into debian
490ecd4 
    ## 0.11.4 (Mar 18, 2019) -- get-ready

    Largely a bug fix release with a few enhancements

    ### Important

    - 0.11.x series will be the last one with support for direct mode of [git-annex][]
      which is used on crippled (no symlinks and no locking) filesystems.
      v7 repositories should be used instead.

    ### Fixes

    - Extraction of .gz files is broken without p7zip installed.  We now
      abort with an informative error in this situation.  ([#3176][])

    - Committing failed in some cases because we didn't ensure that the
      path passed to `git read-tree --index-output=...` resided on the
      same filesystem as the repository.  ([#3181][])

    - Some pointless warnings during metadata aggregation have been
      eliminated.  ([#3186][])

    - With Python 3 the LORIS token authenticator did not properly decode
      a response ([#3205][]).

    - With Python 3 downloaders unnecessarily decoded the response when
      getting the status, leading to an encoding error.  ([#3210][])

    - In some cases, our internal command Runner did not adjust the
      environment's `PWD` to match the current working directory specified
      with the `cwd` parameter.  ([#3215][])

    - The specification of the pyliblzma dependency was broken.  ([#3220][])

    - [search] displayed an uninformative blank log message in some
      cases.  ([#3222][])

    - The logic for finding the location of the aggregate metadata DB
      anchored the search path incorrectly, leading to a spurious warning.
      ([#3241][])

    - Some progress bars were still displayed when stdout and stderr were
      not attached to a tty.  ([#3281][])

    - Check for stdin/out/err to not be closed before checking for `.isatty`.
      ([#3268][])

    ### Enhancements and new features

    - Creating a new repository now aborts if any of the files in the
      directory are tracked by a repository in a parent directory.
      ([#3211][])

    - [run] learned to replace the `{tmpdir}` placeholder in commands with
      a temporary directory.  ([#3223][])

    - [duecredit][] support has been added for citing DataLad itself as
      well as datasets that an analysis uses.  ([#3184][])

    - The `eval_results` interface helper unintentionally modified one of
      its arguments.  ([#3249][])

    - A few DataLad constants have been added, changed, or renamed ([#3250][]):
      - `HANDLE_META_DIR` is now `DATALAD_DOTDIR`.  The old name should be
         considered deprecated.
      - `METADATA_DIR` now refers to `DATALAD_DOTDIR/metadata` rather than
        `DATALAD_DOTDIR/meta` (which is still available as
        `OLDMETADATA_DIR`).
      - The new `DATASET_METADATA_FILE` refers to `METADATA_DIR/dataset.json`.
      - The new `DATASET_CONFIG_FILE` refers to `DATALAD_DOTDIR/config`.
      - `METADATA_FILENAME` has been renamed to `OLDMETADATA_FILENAME`.

* tag '0.11.4': (82 commits)
  Updated CHANGELOG.md for having merged check for not being closed
  [DATALAD RUNCMD] CHANGELOG: Re-linkify 0.11.4 entries
  CHANGELOG.md: Update 0.11.4 entries
  CHANGELOG.md: Adjust the format of a link
  BF: split lines on spaces and commas befoe doing sed capture of #issue
  RF: adjust tools/link_issues_CHANGELOG to have MD links as [#issue][]
  BF+DOC: Fix all links to mark them valid markdown
  BF: Fix markup bug that prevents sphinx-build from succeeding
  DOC: extend coumentation of is_interactive
  BF: guard .istty with try/except
  ENH: ui: Drop progress bars if not attached to tty
  ENH: ui: Add another name for SilentConsoleLog
  BF: check for stdin/out/err to not be closed before checking for .isatty
  RF: HANDLE_META_DIR -> DATALAD_DOTDIR (but keeping an alias for compatibility)
  RF: define/use consts DATASET_CONFIG_FILE, DATASET_METADATA_FILE, METADATA_DIR
  RF: METADATA_DIR/FILE -> OLDMETADATA_DIR/FILE
  BF: do not cause a side-effect on kwargs in @eval_results
  BF: join with ds.path when trying to see if any other metadata file is available
  RF: Move duecredit into its own section so it is not a part of install_requires
  [DATALAD RUNCMD] CHANGELOG: Re-linkify 0.11.4 entries
  ...
yarikoptic added a commit to yarikoptic/datalad that referenced this issue Apr 6, 2019
@yarikoptic
Merge tag '0.11.4' into enh-merge-0.11.x
51cdbe4 
    ## 0.11.4 (Mar 18, 2019) -- get-ready

    Largely a bug fix release with a few enhancements

    ### Important

    - 0.11.x series will be the last one with support for direct mode of [git-annex][]
      which is used on crippled (no symlinks and no locking) filesystems.
      v7 repositories should be used instead.

    ### Fixes

    - Extraction of .gz files is broken without p7zip installed.  We now
      abort with an informative error in this situation.  ([datalad#3176][])

    - Committing failed in some cases because we didn't ensure that the
      path passed to `git read-tree --index-output=...` resided on the
      same filesystem as the repository.  ([datalad#3181][])

    - Some pointless warnings during metadata aggregation have been
      eliminated.  ([datalad#3186][])

    - With Python 3 the LORIS token authenticator did not properly decode
      a response ([datalad#3205][]).

    - With Python 3 downloaders unnecessarily decoded the response when
      getting the status, leading to an encoding error.  ([datalad#3210][])

    - In some cases, our internal command Runner did not adjust the
      environment's `PWD` to match the current working directory specified
      with the `cwd` parameter.  ([datalad#3215][])

    - The specification of the pyliblzma dependency was broken.  ([datalad#3220][])

    - [search] displayed an uninformative blank log message in some
      cases.  ([datalad#3222][])

    - The logic for finding the location of the aggregate metadata DB
      anchored the search path incorrectly, leading to a spurious warning.
      ([datalad#3241][])

    - Some progress bars were still displayed when stdout and stderr were
      not attached to a tty.  ([datalad#3281][])

    - Check for stdin/out/err to not be closed before checking for `.isatty`.
      ([datalad#3268][])

    ### Enhancements and new features

    - Creating a new repository now aborts if any of the files in the
      directory are tracked by a repository in a parent directory.
      ([datalad#3211][])

    - [run] learned to replace the `{tmpdir}` placeholder in commands with
      a temporary directory.  ([datalad#3223][])

    - [duecredit][] support has been added for citing DataLad itself as
      well as datasets that an analysis uses.  ([datalad#3184][])

    - The `eval_results` interface helper unintentionally modified one of
      its arguments.  ([datalad#3249][])

    - A few DataLad constants have been added, changed, or renamed ([datalad#3250][]):
      - `HANDLE_META_DIR` is now `DATALAD_DOTDIR`.  The old name should be
         considered deprecated.
      - `METADATA_DIR` now refers to `DATALAD_DOTDIR/metadata` rather than
        `DATALAD_DOTDIR/meta` (which is still available as
        `OLDMETADATA_DIR`).
      - The new `DATASET_METADATA_FILE` refers to `METADATA_DIR/dataset.json`.
      - The new `DATASET_CONFIG_FILE` refers to `DATALAD_DOTDIR/config`.
      - `METADATA_FILENAME` has been renamed to `OLDMETADATA_FILENAME`.

* tag '0.11.4':
  Updated CHANGELOG.md for having merged check for not being closed
  [DATALAD RUNCMD] CHANGELOG: Re-linkify 0.11.4 entries
  CHANGELOG.md: Update 0.11.4 entries
  CHANGELOG.md: Adjust the format of a link
  BF: split lines on spaces and commas befoe doing sed capture of #issue
  RF: adjust tools/link_issues_CHANGELOG to have MD links as [#issue][]
  BF+DOC: Fix all links to mark them valid markdown
  BF: Fix markup bug that prevents sphinx-build from succeeding
  DOC: extend coumentation of is_interactive
  BF: guard .istty with try/except
  BF: check for stdin/out/err to not be closed before checking for .isatty
  [DATALAD RUNCMD] CHANGELOG: Re-linkify 0.11.4 entries
  CHANGELOG: Add entries for 0.11.4
  ENH: version boost and initial changes to CHANGELOG
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kyleam kyleam

Labels
fix-implemented A fix is available, but has not been merged or released, yet. severity-normal standard severity
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@yarikoptic @kyleam @vassiki