Implement feature to encode and retrieve signac elements by URI. #189

csadorf · 2019-05-19T02:10:47Z

Description

This feature implements the ability to encode certain signac elements as a URI which can then be used to retrieve said element with the signac.open() function.

Minimal example:

>>> uri = project.find_jobs({'foo' 0, 'doc.bar': True}).to_uri()
>>> print(uri)
signac://localhost/Users/csadorf/my_project/api/v1/find?sp.foo=0&doc.bar=True
>>> signac.open(uri)
signac.contrib.project.JobsCursor({'project': 'project', 'filter': '{'sp.foo': 0, 'doc.bar': 'True'}'})

Related to issue #96.

Motivation and Context

This change is motivated by the desire to reliably and unambiguously encode signac elements as unique text-strings. It is much easier to copy & paste such a URI and store it somewhere, instead of awkwardly JSON-encoded filter for example.

Types of Changes

Documentation update
Bug fix
New feature
Breaking change¹

¹The change breaks (or has the potential to break) existing functionality.

Checklist:

I am familiar with the Contributing Guidelines.
I agree with the terms of the Contributor Agreement.
My name is on the list of contributors.
My code follows the code style guideline of this project.
The changes introduced by this pull request are covered by existing or newly introduced tests.

If necessary:

I have updated the API documentation as part of the package doc-strings.
I have created a separate pull request to update the framework documentation on signac-docs and linked it here.
I have updated the changelog.

codecov · 2019-05-19T14:24:31Z

Codecov Report

Merging #189 into feature/integrated-queries will decrease coverage by 1.24%.
The diff coverage is 37.17%.

@@                      Coverage Diff                       @@
##           feature/integrated-queries     #189      +/-   ##
==============================================================
- Coverage                       65.03%   63.79%   -1.25%     
==============================================================
  Files                              37       38       +1     
  Lines                            5554     5626      +72     
==============================================================
- Hits                             3612     3589      -23     
- Misses                           1942     2037      +95

Impacted Files	Coverage Δ
signac/__init__.py	`100% <100%> (ø)`	⬆️
signac/contrib/filterparse.py	`67.59% <26.08%> (-10.06%)`	⬇️
signac/uri.py	`30.43% <30.43%> (ø)`
signac/contrib/project.py	`87.29% <48.27%> (-2.89%)`	⬇️
signac/contrib/job.py	`85.82% <50%> (-4.65%)`	⬇️
signac/core/json.py	`81.57% <0%> (-13.16%)`	⬇️
signac/common/errors.py	`72.22% <0%> (-11.12%)`	⬇️
signac/contrib/indexing.py	`73.77% <0%> (-1.48%)`	⬇️
signac/core/jsondict.py	`93.86% <0%> (-1.23%)`	⬇️
... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8487e60...e67ddd4. Read the comment docs.

codecov · 2019-05-19T14:24:31Z

Codecov Report

Merging #189 into feature/integrated-queries will increase coverage by 0.17%.
The diff coverage is 80.6%.

@@                      Coverage Diff                       @@
##           feature/integrated-queries     #189      +/-   ##
==============================================================
+ Coverage                        65.1%   65.28%   +0.17%     
==============================================================
  Files                              37       38       +1     
  Lines                            5574     5657      +83     
==============================================================
+ Hits                             3629     3693      +64     
- Misses                           1945     1964      +19

Impacted Files	Coverage Δ
signac/__main__.py	`0% <0%> (ø)`	⬆️
signac/contrib/import_export.py	`84.93% <100%> (ø)`	⬆️
signac/contrib/schema.py	`89.16% <100%> (ø)`	⬆️
signac/contrib/linked_view.py	`85.71% <100%> (ø)`	⬆️
signac/__init__.py	`100% <100%> (ø)`	⬆️
signac/contrib/job.py	`90.15% <50%> (-0.32%)`	⬇️
signac/uri.py	`78.26% <78.26%> (ø)`
signac/contrib/project.py	`89.91% <84.09%> (-0.18%)`	⬇️
signac/contrib/filterparse.py	`84.8% <89.87%> (+3.35%)`	⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 99a5847...8750e88. Read the comment docs.

csadorf · 2019-05-19T22:11:33Z

@bdice Is this something that we can integrate with dashboard?

* Keep signac shell command history on a per-project basis. Resolves issue #134. * Update changelog. * Fix Py27 incompatibility. * Readline module not fully implemented for PyPy.

This patch enables the search for jobs with a statepoint- and document- combined filter. To specify whether a key is a statepoint- or a document-key use prefixes 'sp.' and 'doc.' respectively. No prefix is equivalent to the 'sp.' prefix. This patch is backwards compatible with the exception that the index scheme was slightly modified.

To canonicalize all job-find filters at the JobsCursor stage.

bdice · 2019-05-29T02:09:22Z

@bdice Is this something that we can integrate with dashboard?

Probably, but I'm not sure if it even makes sense if the URIs require absolute paths. Is there a "relative path" variant of this planned? Like a Project.open(...)?

csadorf · 2019-06-07T14:58:26Z

@bdice Is this something that we can integrate with dashboard?

Probably, but I'm not sure if it even makes sense if the URIs require absolute paths. Is there a "relative path" variant of this planned? Like a Project.open(...)?

Yes, Project.open(...) is already implemented as part of this PR.

mikemhenry · 2020-04-08T20:32:24Z

I think this will be a really great feature, I've added it to the milestone for v1.5

mikemhenry · 2020-04-08T20:33:29Z

signac/contrib/filterparse.py

 from ..core import json
+from ..common import six


We can get rid of the py2/3 boiler plate

vyasr · 2020-04-25T22:18:19Z

We should revisit this PR after ~~#188~~ #332 is merged, it is pretty close to complete. As discussed with @csadorf, it might be beneficial to get this reviewed by someone more experienced with working with RESTful APIs (i.e. a web developer) to make sure that the URIs we're proposing make sense. @bdice @mikemhenry in any case we should plan to discuss plans for this PR at the dev meeting on Monday.

csadorf · 2020-04-28T13:37:14Z

Brief note: We should probably directly support IRIs as a superset of URIs.

csadorf · 2021-03-25T15:38:53Z

@vyasr This is no longer blocked as #332 is merged, correct?

vyasr · 2021-03-25T21:17:12Z

Correct, this is no longer blocked. However, at this point I think we should plan this as a 2.1 feature so that we can focus our immediate efforts on finishing up the changes required for 2.0. I think this is the next major feature on the docket, though.

stale · 2021-06-02T16:13:46Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

tcmoore3 · 2021-08-10T13:42:18Z

@vyasr and @csadorf, is this still more appropriate as a 2.1 feature, or can it go into 2.0?

csadorf · 2021-08-10T13:50:02Z

I don't know how much more work needs to be done here. I'd leave it up to the judgement of @vyasr .

vyasr · 2021-09-29T02:53:36Z

Sorry for not getting back to this any sooner. I think this is still more appropriate for 2.1. We have enough to get done to get us to a clean state for 2.0 that we'd be better served getting our ducks in line for that and then coming back to this in 2.1.

csadorf requested review from bdice, mikemhenry and a team May 19, 2019 02:11

csadorf force-pushed the feature/uris branch from e67ddd4 to 901a592 Compare May 19, 2019 16:00

csadorf force-pushed the feature/integrated-queries branch from 6df43ad to 99a5847 Compare May 19, 2019 20:03

csadorf force-pushed the feature/uris branch from 901a592 to 064c698 Compare May 19, 2019 20:03

csadorf added 10 commits May 21, 2019 12:32

Keep signac shell command history on a per-project basis. (#194)

bbf2b74

* Keep signac shell command history on a per-project basis. Resolves issue #134. * Update changelog. * Fix Py27 incompatibility. * Readline module not fully implemented for PyPy.

A JobsCursor always uses an integrated filter.

142f927

To canonicalize all job-find filters at the JobsCursor stage.

Extend unit tests to cover integrated and mixed filters.

b4b2ca3

Fix bugs with respect to mixed and integrated filters.

34260b7

Fix bug in unit test implementation.

0344d9f

Implement feature to encode and retrieve signac elements by URI.

d1b3c9c

Fix Py27 incompatibility.

5f96a92

Improve filter encoding and decoding and add related unit tests.

30eb14c

Revise Project.__eq__() function.

8750e88

csadorf force-pushed the feature/uris branch from 0dc54fa to 8750e88 Compare May 25, 2019 15:08

csadorf removed request for a team, bdice and mikemhenry July 5, 2019 14:02

b-butler added the enhancement New feature or request label Jul 5, 2019

b-butler added this to the v1.2.0 milestone Jul 5, 2019

csadorf modified the milestones: v1.2.0, v1.3.0 Jul 5, 2019

csadorf assigned bdice Jul 5, 2019

bdice added this to the v1.4.0 milestone Dec 6, 2019

bdice removed their assignment Jan 6, 2020

bdice removed this from the v1.4.0 milestone Feb 27, 2020

csadorf mentioned this pull request Mar 2, 2020

Proposal: Unify dict classes and improve buffering and synchronization #249

Closed

mikemhenry added this to the v1.5.0 milestone Apr 8, 2020

mikemhenry reviewed Apr 8, 2020

View reviewed changes

signac/contrib/filterparse.py

from ..core import json

from ..common import six

Copy link

Contributor

mikemhenry Apr 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can get rid of the py2/3 boiler plate

csadorf added the blocked Dependent on something else label Apr 26, 2020

bdice modified the milestones: v1.5.0, v1.6.0 Jun 25, 2020

vyasr modified the milestones: v1.6.0, v2.0.0 Sep 16, 2020

vyasr mentioned this pull request Sep 16, 2020

Proposal: Cross-references of jobs #96

Open

vyasr mentioned this pull request Feb 26, 2021

Deprecate doc_filter parameters. #516

Merged

12 tasks

vyasr removed the blocked Dependent on something else label Mar 25, 2021

csadorf modified the milestones: v2.0.0, v2.1.0 Mar 26, 2021

stale bot added the stale label Jun 2, 2021

csadorf added the pinned Instructs stale bot to ignore this issue. label Jun 3, 2021

stale bot removed the stale label Jun 3, 2021

b-butler added the good first issue Good for newcomers label Feb 13, 2023

bdice removed this from the v2.1.0 milestone Jul 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement feature to encode and retrieve signac elements by URI. #189

Implement feature to encode and retrieve signac elements by URI. #189

csadorf commented May 19, 2019 •

edited

codecov bot commented May 19, 2019

codecov bot commented May 19, 2019 •

edited

csadorf commented May 19, 2019

bdice commented May 29, 2019

csadorf commented Jun 7, 2019

mikemhenry commented Apr 8, 2020

mikemhenry Apr 8, 2020

vyasr commented Apr 25, 2020 •

edited

csadorf commented Apr 28, 2020

csadorf commented Mar 25, 2021

vyasr commented Mar 25, 2021

stale bot commented Jun 2, 2021

tcmoore3 commented Aug 10, 2021

csadorf commented Aug 10, 2021

vyasr commented Sep 29, 2021

Implement feature to encode and retrieve signac elements by URI. #189

Are you sure you want to change the base?

Implement feature to encode and retrieve signac elements by URI. #189

Conversation

csadorf commented May 19, 2019 • edited

Description

Motivation and Context

Types of Changes

Checklist:

codecov bot commented May 19, 2019

Codecov Report

codecov bot commented May 19, 2019 • edited

Codecov Report

csadorf commented May 19, 2019

bdice commented May 29, 2019

csadorf commented Jun 7, 2019

mikemhenry commented Apr 8, 2020

mikemhenry Apr 8, 2020

Choose a reason for hiding this comment

vyasr commented Apr 25, 2020 • edited

csadorf commented Apr 28, 2020

csadorf commented Mar 25, 2021

vyasr commented Mar 25, 2021

stale bot commented Jun 2, 2021

tcmoore3 commented Aug 10, 2021

csadorf commented Aug 10, 2021

vyasr commented Sep 29, 2021

csadorf commented May 19, 2019 •

edited

codecov bot commented May 19, 2019 •

edited

vyasr commented Apr 25, 2020 •

edited