cmd/upspin: add -overwrite flag to upspin cp#516
Closed
Conversation
Rewrite the marshaling code to keep track of the size of the messages it is creating, and avoid growing the slice bigger than can be handled on a 32-bit machine. Add tests to verify the conditions are met. Unmarshaling is still not done. Update #405. Change-Id: I46692fcb212d98682b1a562f6eed4eee6ca9314d Reviewed-on: https://upspin-review.googlesource.com/11900 Reviewed-by: Andrew Gerrand <adg@golang.org>
Add these names, which are copies of the Cache and ErrCacheMiss names from the golang.org/x/crypto/acme/autocert package, so that users of the uspin.io/cloud/https package need not import their own version of the autocert package, rather than the version vendored by this repository. See the documentation for ErrAutocertCacheMiss for more details. This is a somewhat awkard work-around the issue that upspin.io vendors its dependencies and gcp.upspin.io does not. These types will stay around until we have fully resolved issue #214. Change-Id: I61d4ab3201d9da8b9bb57d30531a0000afb2bad7 Reviewed-on: https://upspin-review.googlesource.com/11920 Reviewed-by: Rob Pike <r@golang.org>
The fast path for fetching objects by HTTPS was not returning a structured errors.NotExist error on a 404. Fix that. Change-Id: I5fd18e2974583d802e04cbebac8ff7af66e4c9a8 Reviewed-on: https://upspin-review.googlesource.com/12020 Reviewed-by: Rob Pike <r@golang.org>
This is a command, so instead of saying "upspinfs/fs.Write" it's OK just to say "Write". A tentative step towards establishing some rules about op strings (see issue 447). Update #447 Update #212 Change-Id: Ib53218710bc9afa38a57c6c1094cb76758245396 Reviewed-on: https://upspin-review.googlesource.com/12021 Reviewed-by: Andrew Gerrand <adg@golang.org>
It's an internal package and shouldn't be displayed to users when something goes wrong. This should reduce the noise a little when errors happen. If we need to know the full stack trace to debug a problem, we can build with -tags debug. Update #212. Change-Id: Ieedaa9a55805bb7a19e42f0562d7c5a31d4aacc2 Reviewed-on: https://upspin-review.googlesource.com/12041 Reviewed-by: Andrew Gerrand <adg@golang.org>
…method Also fix a typo in pack/ee. Update #212. Change-Id: Ia061c52a9cfdf22dd6dc8f5451ed245092a81db3 Reviewed-on: https://upspin-review.googlesource.com/12040 Reviewed-by: Andrew Gerrand <adg@golang.org>
Change-Id: I5c47a6e29e797d336e18a7d44c7c5b2c77d28fec Reviewed-on: https://upspin-review.googlesource.com/12060 Reviewed-by: Rob Pike <r@golang.org>
The previous version was inscrutable and was not updated when sig2 was added to packdata. Change-Id: I22eee729dec26ab55dcb97191935fdb0f527abf7 Reviewed-on: https://upspin-review.googlesource.com/12080 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Rob Pike <r@golang.org>
Protects against overflow when unpacking large items on 32-bit machines. Fixes #405. Change-Id: I6f962ce11963e89cc857da3445607f36602b1c2d Reviewed-on: https://upspin-review.googlesource.com/12100 Reviewed-by: Andrew Gerrand <adg@golang.org>
There were only two "usage examples" and both serve better as FAQ entries. Transfer the information to the FAQ, with some rewriting, and delete the examples file. Change-Id: Icfa4ac19f7637f2f615d13252bf88f96ce9c01f3 Reviewed-on: https://upspin-review.googlesource.com/12140 Reviewed-by: Andrew Gerrand <adg@golang.org>
A couple of things went wrong in commit fa5dd30. First, the new rpc/local.CacheEndpoint function didn't return the correct endpoint for the local cache. It should have had a "remote," prefix and a ":80" suffix. Second, I left the config.SetCacheEndpoint function in by mistake (which in that change became a no-op, as it was replaced by the generic config.SetValue), and the cacheserver was using that function to prevent the cacheserver from calling itself. Both these issues are fixed by this change. Fix #443 Change-Id: I74bacc993c86383807120c337700c607241a764f Reviewed-on: https://upspin-review.googlesource.com/12160 Reviewed-by: Rob Pike <r@golang.org>
When we replaced config.SetCacheEndpoint with the general config.SetValue, we didn't preserve the behavior where a specific cache endpoint could be provided in the config (it just assumed it would be "yes" or "no"). The cacheserver tests actually depend on being able to set the cache endpoint specifically, so this change restores that behavior and fixes the test. Second to this, the cacheserver end-to-end tests didn't actually fail with an invalid cache endpoint specified. This is because the store/remote and dir/remote packages would silently ignore a failure to connect to a cacheserver. It may be more user-friendly in a short term sense for a broken cacheserver to be bypassed silently, but I suggest that in the long term it is better that people know whether or not their cacheserver is broken. In that spirit, this change makes store/dirserver remote calls fail if the cacheserver is inaccessible. I believe we would have caught cacheserver-related issues (like #443) much earier had this been the original behavior. Change-Id: I5d660d471dd6142b7b9dd8b2394bf138b5172ab0 Reviewed-on: https://upspin-review.googlesource.com/12161 Reviewed-by: Rob Pike <r@golang.org>
Written mostly be Eric Grosse, edited by me. Change-Id: I847702d34691a5684ba595387ca597068ab18221 Reviewed-on: https://upspin-review.googlesource.com/12180 Reviewed-by: Eric Grosse <grosse@gmail.com>
This lifts the restriction of only serving insecure HTTP on loopback addresses. Instead of exiting it will print a warning message if serving insecure HTTP on a non-loopback address. It enables use cases like running upspin as a Pod in Kubernetes cluster where TLS termination is handled at the ingress level instead of directly in upspin. Fixes #448 Change-Id: I0406512fef99888e731151cf3e340d38c2489d1a Reviewed-on: https://upspin-review.googlesource.com/12200 Reviewed-by: Andrew Gerrand <adg@golang.org>
Change-Id: Idca6d4016e35139532addc3065d2a02741da6e09 Reviewed-on: https://upspin-review.googlesource.com/12240 Reviewed-by: Rob Pike <r@golang.org>
Need some fancy scaffolding to test, so this is untested. We might one day create such a thing, a way to program a misbehaving server for package-level tests. But not now. Fixes #451. Change-Id: Ib83ac2785921342dfb912af2f150eef7c3ee4b9e Reviewed-on: https://upspin-review.googlesource.com/12320 Reviewed-by: Andrew Gerrand <adg@golang.org>
Add words at the top level help and also in the help for the shell. Fixes #450. Change-Id: I5aec015aa2613877447356156ce8da612e4e1783 Reviewed-on: https://upspin-review.googlesource.com/12321 Reviewed-by: Andrew Gerrand <adg@golang.org>
Change-Id: I30048aa9469b5179d84ebb8aa2ca5dd0cad3d0b5 Reviewed-on: https://upspin-review.googlesource.com/12460 Reviewed-by: Rob Pike <r@golang.org>
Change-Id: I60351d263cb0da5c39a5c0a1ce0f746537cd689d Reviewed-on: https://upspin-review.googlesource.com/12500 Reviewed-by: Rob Pike <r@golang.org>
Change-Id: I187645bdfe7fb54d6b8dc81cb506ce7af55144a2 Reviewed-on: https://upspin-review.googlesource.com/12540 Reviewed-by: Rob Pike <r@golang.org>
Change-Id: I4e4f84fdc70afe2bd8ea4f15c6d92aeacb7d0369 Reviewed-on: https://upspin-review.googlesource.com/12520 Reviewed-by: Rob Pike <r@golang.org>
The old spec required DirServer.Put to return nil on success, to avoid just returning a huge struct that was just sent by the client. (The dir/server code didn't do this; it was not honoring the spec. We didn't notice because, unlike in inprocess, there were no tests for the property and Client returns an entry anyway, since it's in memory.) To address problems syncing a cache using Watch with Puts that have happened while disconnected, here we change Put to return a DirEntry, but to keep things efficient the returned entry is marked Incomplete and returns just a Sequence Number. This CL updates the DirServer implementations and Client code. The cache is not yet using this new property Update #445. Change-Id: Ic53ec16a26e1fadbd484f1de46e30c069107717c Reviewed-on: https://upspin-review.googlesource.com/12480 Reviewed-by: David Presotto <presotto@gmail.com>
…heir default value Fixes #455 When cacheutil.Start started the cacheserver, it would pass all possible command line flags with their current value. This was confusing to see in a ps when someone set the flag to a different value in their upspin config. The value was neing set correctly inside cacheserver but that wasn't evident. To avoid this, cacheutil.Start now only passes those flags at values other than their default. Change-Id: Ie084e783203512d1063c8d88c421d772db985c0c Reviewed-on: https://upspin-review.googlesource.com/12560 Reviewed-by: Rob Pike <r@golang.org>
Change-Id: I124d8abe3b4aeb3c15cba59f1b7e4980a7589e36 Reviewed-on: https://upspin-review.googlesource.com/12580 Reviewed-by: Rob Pike <r@golang.org>
…comment This was left unaddressed after https://upspin-review.googlesource.com/11741 Change-Id: I24ba40602910028b55038a43a09f3aadb351841e Reviewed-on: https://upspin-review.googlesource.com/12620 Reviewed-by: David Presotto <presotto@gmail.com>
<Cherry-pick from dev.seq to master branch> This was left unaddressed after https://upspin-review.googlesource.com/11741 Change-Id: I24ba40602910028b55038a43a09f3aadb351841e Reviewed-on: https://upspin-review.googlesource.com/12620 Reviewed-by: David Presotto <presotto@gmail.com> Reviewed-on: https://upspin-review.googlesource.com/12640
Allows the changing of the Time field in a DirEntry. In client.SetTime, if the PathName refers to a link, do not follow it but instead change the Time of the link itself. Also change client.Dup and client.Rename link behavior to match. Fixes #440 Fixes #438 Change-Id: I5f6666a31e425511665ff98987eea69120d4b4fd Reviewed-on: https://upspin-review.googlesource.com/12600 Reviewed-by: Rob Pike <r@golang.org> Reviewed-by: Eric Grosse <grosse@gmail.com>
<Cherry-pick from master branch to dev.seq> Allows the changing of the Time field in a DirEntry. In client.SetTime, if the PathName refers to a link, do not follow it but instead change the Time of the link itself. Also change client.Dup and client.Rename link behavior to match. Fixes #440 Fixes #438 Change-Id: I5f6666a31e425511665ff98987eea69120d4b4fd Reviewed-on: https://upspin-review.googlesource.com/12600 Reviewed-by: Rob Pike <r@golang.org> Reviewed-by: Eric Grosse <grosse@gmail.com> Reviewed-on: https://upspin-review.googlesource.com/12660 Reviewed-by: David Presotto <presotto@gmail.com>
The cache was keep track of bytes used for all new storage but not for storage existed before the cache code started. Fixes #452 Change-Id: I802968e9a9e6409aebfd0a81eebd66360dcc666e Reviewed-on: https://upspin-review.googlesource.com/12680 Reviewed-by: Rob Pike <r@golang.org> Reviewed-by: Eric Grosse <grosse@gmail.com>
<Cherry-pick from master branch> The cache was keep track of bytes used for all new storage but not for storage existed before the cache code started. Fixes #452 Change-Id: I802968e9a9e6409aebfd0a81eebd66360dcc666e Reviewed-on: https://upspin-review.googlesource.com/12680 Reviewed-by: Rob Pike <r@golang.org> Reviewed-by: Eric Grosse <grosse@gmail.com> Reviewed-on: https://upspin-review.googlesource.com/12700
…ommand arg fixes #480 Change-Id: I2e634083ceb8db91b665d6aa95f3e4eeeeaeab5c Reviewed-on: https://upspin-review.googlesource.com/16840 Reviewed-by: Rob Pike <r@golang.org> Reviewed-by: Andrew Gerrand <adg@golang.org>
Without this change, a malicious server could cause the client to allocate without bound. Fix #551 Change-Id: I6488cef182307be7eb5b426954e343cacd247f9c Reviewed-on: https://upspin-review.googlesource.com/16860 Reviewed-by: Rob Pike <r@golang.org>
If an init() function spawns a goroutine that logs (like shutdown currently does), there is a race condition between flag-parsing and that logging call. While this is unlikely to cause problems in practice, best be safe and guard global mutable state with a lock. Change-Id: Ied4965f2948e3d1f311f4ddfdb4e66bf575d0591 Reviewed-on: https://upspin-review.googlesource.com/16900 Reviewed-by: Rob Pike <r@golang.org> Reviewed-by: Andrew Gerrand <adg@golang.org>
Change-Id: If65c7793f08075001f990f092987df098b5e82cf Reviewed-on: https://upspin-review.googlesource.com/16940 Reviewed-by: Rob Pike <r@golang.org>
One call to errors.E in client.lookup failed to report the original path name. Provide it so we actually see what the name was that we couldn't find. Change-Id: If2b6f5265a95b67472f4d901f44a3bb75c830555 Reviewed-on: https://upspin-review.googlesource.com/16960 Reviewed-by: Andrew Gerrand <adg@golang.org>
A Watch stream can be significantly delayed presenting the watcher
with events. The result is that local actions can be replayed
as they are received on the Watch interface. This can lead to local
inconsistencies until the Watch stream has caught up with local
changes.
In particular, we were seeing this with a git instance that was
renaming a file and would then see the original name continue to
exist because the events looked like this:
client Server
Put(a)
Put(b) (as a clone of a)
Delete(a)
Event(a created)
Event(b created)
Lookup(a) (returns true when is should have been false)
Event(a deleted)
This CL remembers the sequence numbers for the last 2000 Puts
and ignores any Events with the same or lower sequences
to avoid these replays. We should also keep track of Deletes
but Deletes don't return the sequence number. Luckily
Delete Events just flush information from the cache so that
they do not create inconstencies, just unneeded work if performed
multiple times. If we changed the cache to also remember
deletes, the Delete API will have to change a bit.
This doesn't solve the problem, just reduces it to a hopefully
bearable level. The directory server could delay Watch events
arbitrarily long with > two thousand files being changed before
the Watch Event is delivered for the first file. However, this
would be an extreme case and one might argue that this would
indicate a broken or tremendously overloaded server.
fixes #552
Change-Id: I073dc6478f78771ee7288884c8354ae6749b6ac7
Reviewed-on: https://upspin-review.googlesource.com/16920
Reviewed-by: Rob Pike <r@golang.org>
Reviewed-by: Andrew Gerrand <adg@golang.org>
This makes traces a bit easier to scan. Change-Id: Ic3b856a38f5c288057146de69c7990276b7fa8cd Reviewed-on: https://upspin-review.googlesource.com/16961 Reviewed-by: Rob Pike <r@golang.org> Reviewed-by: David Presotto <presotto@gmail.com>
Instead of Ops like this:
store/remote: "remote,test.example.com:443": store.Get:
Show them like this:
store/remote("test.example.com:443").Get:
Change-Id: Ie6733d4a06352c91c75bdd481318e5e66dc4ec5e
Reviewed-on: https://upspin-review.googlesource.com/16962
Reviewed-by: Rob Pike <r@golang.org>
Change-Id: Iee41eb8d6a3f2a42b7a79d84fb2e3d65a8219d56 Reviewed-on: https://upspin-review.googlesource.com/16963 Reviewed-by: David Presotto <presotto@gmail.com> Reviewed-by: Rob Pike <r@golang.org>
…set time This cropped up in the mv command. It changes the time before closing the newly created file when the file has to be copied. Since we normally don't perform the writeback until the close, this caused the Setattr to return a ENOENT. The fix is to writeback when trying to set the time of a currently open file. I also moved a set of common checks into writeback itself since it was silly to keep copying the checks everywhere writeback is called. Change-Id: I531cc5dd5fe4ac4abef4561b56dfe995607636e2 Reviewed-on: https://upspin-review.googlesource.com/16941 Reviewed-by: Rob Pike <r@golang.org> Reviewed-by: Andrew Gerrand <adg@golang.org>
…ying to set time" This reverts commit ee1e712. Reason for revert: forgot to upload the last changes Change-Id: Ic6995e2951add66c1f8ab8e53ed8158819389d7a Reviewed-on: https://upspin-review.googlesource.com/16980 Reviewed-by: Andrew Gerrand <adg@golang.org>
…set time This cropped up in the mv command. It changes the time before closing the newly created file when the file has to be copied. Since we normally don't perform the writeback until the close, this caused the Setattr to return a ENOENT. The fix is to writeback before trying to set the time of a currently open file. I also moved a set of common checks into writeback itself since it was silly to keep copying the checks everywhere writeback is called. Change-Id: I2625960f77365840b471f9ce869f982e43ce5461 Reviewed-on: https://upspin-review.googlesource.com/17040 Reviewed-by: Rob Pike <r@golang.org>
It's important that the cleanup function sees an accurate view of the world, so that it doesn't try to list or delete files that don't exist. Change-Id: Ic39898967c17aaa2c0f927274e6c48c17927d41e Reviewed-on: https://upspin-review.googlesource.com/17060 Reviewed-by: Eric Grosse <grosse@gmail.com>
I'm no GCP expert, but "gcloud beta" was correcting my spelling this afternoon; may as well fix our docs. Change-Id: I589a295570772ad835f822cab2370bae9687ac9f Reviewed-on: https://upspin-review.googlesource.com/17100 Reviewed-by: Andrew Gerrand <adg@golang.org>
Change-Id: I94b2d7a4bebc4c9354e1c1aa2573291426438a7c Reviewed-on: https://upspin-review.googlesource.com/17080 Reviewed-by: Andrew Gerrand <adg@golang.org>
Add the upspin.ListRefsMetadata reference and the upspin.ListRefsResponse and upspin.ListRefsItem types. Add the storage.Lister interface. Add permission checking of these calls to serverutil/perm. Implement the API in store/server. In the service of garbage collection. Updates #145 Change-Id: I34b8d8644208e3c43b9b46a2f25bc1a0e8943ec6 Reviewed-on: https://upspin-review.googlesource.com/16881 Reviewed-by: David Presotto <presotto@gmail.com> Reviewed-by: Rob Pike <r@golang.org>
Fixes the other half of issue 470. fixes #470 Change-Id: Id170ec9e1b25280b1c2eb2f4c9fbecd7cdc9b56c Reviewed-on: https://upspin-review.googlesource.com/17160 Reviewed-by: Rob Pike <r@golang.org>
The previous behavior was to allow any store writer to list references, which leaks information unnecessarily. Only the storage user (the administrator) has a need to list references, so narrow the access. Change-Id: I923b8cf8dad8b4fac8983740e4baef0452e2fb74 Reviewed-on: https://upspin-review.googlesource.com/17180 Reviewed-by: Rob Pike <r@golang.org>
Change-Id: I8c38286595a660d5f25c616b06317ab375c24e13 Reviewed-on: https://upspin-review.googlesource.com/17220 Reviewed-by: Rob Pike <r@golang.org>
Change-Id: I4ab0557caf356c27c43c3df5b1b8dd8f197289ba Reviewed-on: https://upspin-review.googlesource.com/17221 Reviewed-by: Rob Pike <r@golang.org>
This test is less exhaustive than the tests in the storage implementations themselves, but it suffices to test the passing through of arguments and JSON transcoding. Change-Id: I7957c4d52e475b2ca8c33b0d972724a8ec8ad352 Reviewed-on: https://upspin-review.googlesource.com/17240 Reviewed-by: Rob Pike <r@golang.org>
The current code incremented the sequence number but didn't use it in the watch entry. Change-Id: I8000f31bcd991835eba684e7aaf401e69d494a07 Reviewed-on: https://upspin-review.googlesource.com/17280 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Rob Pike <r@golang.org>
…rrors Since the reference may contain an @ character, that could lead the errors package to misconstrue the provided string as a mistyped upspin.PathName or upspin.UserName. Change-Id: I5626bae3a2c7b6a5aa48fb448786c4b324785ee1 Reviewed-on: https://upspin-review.googlesource.com/17360 Reviewed-by: Rob Pike <r@golang.org>
In preparation for writeback of DirEntries (as opposed to the current writethrough), I've added a check in the Put and Delete RPCs. I don't currently obey the result, just check it against what the server really does to make sure I've got it correct. The next step will be an actual writeback queue. Change-Id: I22a40483f599a51d99ac6bff17e1789235554529 Reviewed-on: https://upspin-review.googlesource.com/17300 Reviewed-by: Andrew Gerrand <adg@golang.org> Reviewed-by: Rob Pike <r@golang.org>
This reverts commit 171824b. Reason for revert: It broke the build Change-Id: I1d5ee861905103e4b76e191f2925e69821a5df92 Reviewed-on: https://upspin-review.googlesource.com/17400 Reviewed-by: Andrew Gerrand <adg@golang.org>
Fixes #479 Change-Id: I9deb58bb0746f19f97be91410885ee2d1dffe0d5 Reviewed-on: https://upspin-review.googlesource.com/17321 Reviewed-by: Rob Pike <r@golang.org>
Use Client's Open and Create methods instead of Get and Put. This effectively limits the peak memory usage of a tar/untar operation to upspin.BlockSize plus epsilon, while before this change you required as much memory as the largest file you were tarring/untarring. Change-Id: Ib7c5a2011e73e3d6089b0df9ad353be231437052 Reviewed-on: https://upspin-review.googlesource.com/17460 Reviewed-by: Rob Pike <r@golang.org>
The default behavior is to overwrite destination files. To skip overwriting existing destination files, set -overwrite=false. Change-Id: Ic7e0b82542f8a5368f7ffa38d9abe44e5868b1c8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automatically created pull request. Do not review or merge this PR.