Storage repository by charlesdaniels · Pull Request #1768 · fyne-io/fyne

charlesdaniels · 2021-01-11T21:01:17Z

This is a continuation of #1665. See also the proposal.

Divergences from the proposal

These are generally discussed in more detail, this section provides a summary and keeps track of them in one place with direct links.

Add repository.ForURI() method
- Name updated from RepositoryForURI() here
Move URIRootError to repository
Add repository.CustomURIRepository extension interface to allow overriding storage.ParseURI() behavior
- Additional noteworthy discussion here.
Move uri and back-end logic for ParseURI and NewFileURI into repository to avoid a circular import, and allow Generic{Parent,Child}() to call ParseURI
CreateListable() has been added to ListableRepository to allow directories to be created

The underlying work to provide implementations of these methods has not been done yet pending discussion, so this will break the build. This also adds Destroy() as described in #1509, though again no implementation is defined yet. This also adds detailed comments to the members of the URI interface, to make it clear what is expected of URI implementations.

This will be immediately reverted, but if we decide to revive it for some reason, it might be a good starting point. I started writing this and then realized Go already has an RFC3986 implementation here: https://golang.org/pkg/net/url/ I plan to use that unless there is some reason to build out or own instead.

This reverts commit adf04ad.

In retrospect, there is a lot of redundancy In the prose of the given implementation. I will do a second pass to tidy it up at some point.

@toaster

Note that I have veered much more towards @toaster 's (Tilo) proposal than I had originally. My change of heart is documented in the relevant proposal here: https://github.com/fyne-io/fyne/wiki/Proposal%3A-The-Fyne-URI-Philosophy I have decided not to implement URINotListable on the basis that URIOperationImpossible already encompasses it's purpose, and additionally I want to encourage people to use Listable() rather than just trying to List() and then checking the error. I decided to move List and Listable() into the Repository interface, though developers can always throw URIOperatioNotSupported if they are impossible to implement. I could be convinced to move them into their own ListableRepository interface to be optionally registered, but to keep them together (since you cannot have one without the other). This was discussed a little bit, but I am on the fence. Since I decided to go with more of Tilo's approach, I did not add Move() or Duplicate(). If folks have opinions on using these names versus Rename() and Copy(), please add them on the PR.

@andydotxyz

Per our meeting today, I am merging the existing work into the new `storage_repository` feature branch so @andydotxyz and I can collaborate more easily. We will open a new PR `storage_repository -> develop` soon. Despite what the GitHub UI may say, this does **not** affect the approval state of this PR as far as the core team is concerned.

charlesdaniels · 2021-01-12T01:20:31Z

As of eb6c773 I have stubbed out everything from the proposal. What remains is:

Implementing the TODOs
Implementing the registration system
Implementing the FileRepository
Implementing a new generic URI implementation that supports the updated URI interface.
Re-working driver code for the desktop and mobile drivers.
Tests!

I had to diverge from the proposal in the following ways:

I have added a RepositoryForURI method to storage/repository which returns the repository instance registered for a given URI. This was needed because other methods in the repository package need to have access to the registered methods for schemes (e.g. GenericCopy()), but methods in the storage package still need to be able to access these methods as well.
I think that we need to be able to return URIRootError from inside of repository. I propose moving it into repository and then changing storage.URIRootError to just be assigned repository.URIRootError, with a marked deprecation. I have not done this, but plan to if nobody has a better idea.

charlesdaniels · 2021-01-12T01:32:00Z

Another case where I needed to differ: when we removed ParseURI() from Repository, it seems we also removed it from storage. To the extent of my understanding, it should still be present in storage, so I have re-added it. Otherwise, there is no entry point for obtaining a new URI.

We also did not discuss if we intended to keep or deprecate NewFileURI(). I think this is likely to be a very common-case thing to want to do, so I motion we keep it in.

charlesdaniels · 2021-01-12T01:53:51Z

I have realized that we never properly discussed what to do with storage.NewURI(). I have tentatively marked it as deprecated, since it cannot report an error.

Most notably, I have renamed RepositoryForURI() to RegisteredRepository(), to placate the linter.

AlbinoGeek · 2021-01-12T02:11:05Z

I feel the divergences are acceptable.

If it was added in 1.4 all we can do is deprecate it, I think?

Otherwise, yes, I (as a Developer) one way to Create (New) and one way to Get (Parse) a URI.

So far I'm happy with your approaches / mediations.

charlesdaniels · 2021-01-12T02:13:56Z

If it was added in 1.4 all we can do is deprecate it, I think?

AFAIK, policy is that it stays in for at least one major release.

So far I'm happy with your approaches / mediations.

Minor note - I changed RepositoryForURI() because it made the linter mad, so it's now RegisteredRepository(). Suggestions welcome.

andydotxyz · 2021-01-12T14:16:30Z

NewFileURI is indeed very common. It delegates to NewURI.
I cannot remember what ParseURI erros were expected to be, gven that we are offering basically no guarantees in the URI implementation with regards to its suitability for the repository it relates to...

I don't think we can move NewFileURI to ParseFileURI as that isn't really what it's doing.

andydotxyz · 2021-01-12T14:17:51Z

Minor note - I changed RepositoryForURI() because it made the linter mad, so it's now RegisteredRepository(). Suggestions welcome.

What was the error? I don't think RegisteredRepository really describes the action. RepositoryForScheme would be more accurate.

stuartmscott

The comment to code ratio is incredible!

Added a few comments of items that are more obvious to me from reading this PR than were from reading the wiki spec.

Great work Charles!

charlesdaniels · 2021-01-12T17:04:07Z

@andydotxyz

I don't think we can move NewFileURI to ParseFileURI as that isn't really what it's doing.

It also handles back-slashes on Windows, which we probably want to keep working for usability even if it's not in the spec.

What was the error? I don't think RegisteredRepository really describes the action. RepositoryForScheme would be more accurate

repository.Repository... "stutters" according to the linter. If you are OK with it and know how to disable linting for that method, I think it was a better name. (edit for posterity, here is a deep link to the exact error)

stuartmscott · 2021-01-12T17:17:00Z

What about repository.Lookup(string)?

charlesdaniels · 2021-01-17T22:22:13Z

Consider testing for infinite recursion in symlinks. Most *mix tools are intelligent enough to give up once they hit a recursive link.

(context)

@AlbinoGeek

I don't think this is a significant risk, since we don't ever recurse, other than I guess in file dialog where the user presumably knows what's going on.

As it stands, FileRepository follows the default simlink handling behavior of the standard library. I trust the Go devs to have non-stupid defaults.

I think this is a good thing to have in mind, but I view this as a path normalization thing, which we agreed to punt to the future.

charlesdaniels · 2021-01-17T22:26:10Z

Decision on Directory Creation (`CreateListable()`)

We realized that creating directories was impossible with the API as specified. This was discussed in this thread. As a result, we decided to add CreateListable(u fyne.URI) error to the ListableRepository interface.

We also considered creating a new interface (e.g. ListableCreatorRepository), or adding it to WriteableRepository (as it technically is performing a write operation), but decided against both cases.

We note that implementations may return ErrOperationNotSupported in the event that the repository is read-only but is still list-able, e.g. in some hypothetical zipfs:// repository.

The relevant changes were added in fa994f8 and tests added in 03aca12.

charlesdaniels · 2021-01-17T23:05:32Z

I suspect you are correct. I knew we would need to do our own parser, or find one, but I had hoped to kick the can down the road. If using ttacon/uri makes the tests pass, I have no issue with using it as an internal-only dependency. At a skim, it looks decently written.

(context)

@andydotxyz

I think your issue should be fixed. As of d95d323 I am using fredbi/uri for URI parsing. However, I am using an ugly hack to get it to correctly handle cases like foo:///some/path, since it thinks the host is empty and throws an error.

I will raise the issue upstream and see what the author of the parser says. RFC396 uses this style of URI as an example explicitly (IETF RFC3986, §1.1 pp. 5).

Edit: issue opened upstream.

stuartmscott

PR is looking good, only one unresolved comment remaining

charlesdaniels · 2021-01-18T01:24:33Z

@stuartmscott I think this is the only remaining comment thread that is open. It is solved to my satisfaction unless you had something else you wanted to discuss on that front?

stuartmscott

@charlesdaniels what about #1768 (review)?

Co-authored-by: Stuart Scott <stuart.murray.scott@gmail.com>

stuartmscott

Missed a couple, but otherwise this PR looks good to go and should be able to merge tonight

stuartmscott · 2021-01-18T02:10:15Z

I think the widget.Form failures are flakey, can you try re-run them?

--- FAIL: TestForm_Hints (0.21s)
    util.go:47: 
        	Error Trace:	util.go:47
        	            				test.go:38
        	            				form_test.go:151
        	Error:      	Not equal: 
        	Test:       	TestForm_Hints
        	Messages:   	Image did not match master. Actual image written to file://D:\a\fyne\fyne\widget\testdata\failed\form\hint_initial.png.
    util.go:47: 
        	Error Trace:	util.go:47
        	            				test.go:38
        	            				form_test.go:154
        	Error:      	Not equal: 
        	Test:       	TestForm_Hints
        	Messages:   	Image did not match master. Actual image written to file://D:\a\fyne\fyne\widget\testdata\failed\form\hint_invalid.png.
    util.go:47: 
        	Error Trace:	util.go:47
        	            				test.go:38
        	            				form_test.go:157
        	Error:      	Not equal: 
        	Test:       	TestForm_Hints
        	Messages:   	Image did not match master. Actual image written to file://D:\a\fyne\fyne\widget\testdata\failed\form\hint_valid.png.

charlesdaniels · 2021-01-18T02:26:50Z

You OK if I merge @stuartmscott ?

stuartmscott · 2021-01-18T02:27:28Z

Go ahead!

charlesdaniels · 2021-01-18T04:20:08Z

+
+	// There was no repository registered, or it did not provide a parser
+
+	// Ugly hack to work around fredbi/uri. Technically, something like


NOTE: forgot to update this comment, but actually foo:///bar/baz is totally valid and should give us /bar/baz as the path. This is documented in the discussion thread on #1768.

charlesdaniels and others added 12 commits December 17, 2020 15:14

add Since: comments

e904cf8

Merge branch 'develop' into uri_rework

e998ab1

Revert "add storage/rfc3986"

1017633

This reverts commit adf04ad.

first pass at Repository interface

c363bbc

In retrospect, there is a lot of redundancy In the prose of the given implementation. I will do a second pass to tidy it up at some point.

add stuarts suggestions

c764b37

remove functions from URI interface

d6608cb

add RFC3986 methods to the URI interface

a68a6b2

set up new Repository interfaces

22ac811

charlesdaniels assigned andydotxyz and charlesdaniels Jan 11, 2021

write stubs described in the proposal

eb6c773

add storage.ParseURI()

2ccf744

bring storage.uri up to compliance with fyne.URI

a1f5bf8

charlesdaniels added 2 commits January 11, 2021 20:57

fix goimports checks

724aeeb

fix linter errors

d29bd29

Most notably, I have renamed RepositoryForURI() to RegisteredRepository(), to placate the linter.

add test cases for the new URI methods

21953ca

stuartmscott reviewed Jan 12, 2021

View reviewed changes

Comment thread storage/repository/repository.go

Comment thread storage/repository/repository.go Outdated

Comment thread storage/repository/repository.go

charlesdaniels added 3 commits January 17, 2021 17:07

add CreateListable tests

03aca12

text that Writer() errors for non-existant parent

635f0bf

use NewFileURI in repository/file.go

c294f16

add fredbi/uri as the URI parser

d95d323

stuartmscott reviewed Jan 17, 2021

View reviewed changes

Comment thread internal/driver/gomobile/repository.go

charlesdaniels and others added 3 commits January 17, 2021 18:23

add type assertions to gomobile/repository.go

a4aa551

return ErrOperationNotSupported instead

c65a897

Mobile does not yet have all of these functions

b1ed0db

andydotxyz previously approved these changes Jan 17, 2021

View reviewed changes

stuartmscott suggested changes Jan 18, 2021

View reviewed changes

Comment thread internal/repository/file.go Outdated

Comment thread storage/repository/repository.go Outdated

Comment thread internal/repository/memory.go Outdated

add ForScheme

184d27b

charlesdaniels dismissed andydotxyz’s stale review via 184d27b January 18, 2021 01:42

charlesdaniels and others added 3 commits January 17, 2021 20:43

improve docs for NewFileRepository

5aa3556

Co-authored-by: Stuart Scott <stuart.murray.scott@gmail.com>

s/Writeable/Writable/g

cf845be

wording nit

35ed6d4

stuartmscott suggested changes Jan 18, 2021

View reviewed changes

Comment thread internal/repository/file.go Outdated

Comment thread storage/repository/generic.go Outdated

s/writeable/writable/g

bafe9c8

stuartmscott approved these changes Jan 18, 2021

View reviewed changes

Merge branch 'develop' into storage_repository

8dc392c

charlesdaniels merged commit 7abf690 into develop Jan 18, 2021

charlesdaniels commented Jan 18, 2021

View reviewed changes

andydotxyz deleted the storage_repository branch April 7, 2021 14:07


		// There was no repository registered, or it did not provide a parser

		// Ugly hack to work around fredbi/uri. Technically, something like

Uh oh!

Uh oh!

Conversation

charlesdaniels commented Jan 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Divergences from the proposal

Uh oh!

charlesdaniels commented Jan 12, 2021

Uh oh!

charlesdaniels commented Jan 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

charlesdaniels commented Jan 12, 2021

Uh oh!

AlbinoGeek commented Jan 12, 2021

Uh oh!

charlesdaniels commented Jan 12, 2021

Uh oh!

andydotxyz commented Jan 12, 2021

Uh oh!

andydotxyz commented Jan 12, 2021

Uh oh!

stuartmscott left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

charlesdaniels commented Jan 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stuartmscott commented Jan 12, 2021

Uh oh!

charlesdaniels commented Jan 17, 2021

Uh oh!

charlesdaniels commented Jan 17, 2021

Decision on Directory Creation (CreateListable())

Uh oh!

charlesdaniels commented Jan 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

stuartmscott left a comment

Choose a reason for hiding this comment

Uh oh!

charlesdaniels commented Jan 18, 2021

Uh oh!

stuartmscott left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stuartmscott left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

stuartmscott commented Jan 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

charlesdaniels commented Jan 18, 2021

Uh oh!

stuartmscott commented Jan 18, 2021

Uh oh!

charlesdaniels Jan 18, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

charlesdaniels commented Jan 11, 2021 •

edited

Loading

charlesdaniels commented Jan 12, 2021 •

edited

Loading

charlesdaniels commented Jan 12, 2021 •

edited

Loading

Decision on Directory Creation (`CreateListable()`)

charlesdaniels commented Jan 17, 2021 •

edited

Loading

stuartmscott commented Jan 18, 2021 •

edited

Loading