Property implementation #8224

paulidale · 2019-02-13T11:15:13Z

A first pass at implementing some efficient property definition and query infrastructure.

Properties are defined by providers as strings which are parsed into name, value pairs.
Queries are parsed into name relation value triplets.
Implementations that match the query are considered for further use whereas implementations that don't match are immediately discarded from consideration.

The property system, allows global properties to be specified in addition to local properties (which can override globals).

If there is a match, there are capabilities for caching the outcome which will improve performance in the future.

The implementation database permits several interfaces to operate within:

add,
remove,
fetch,
cache_get,
cache_set and
set_global_properties

The generate usage pattern is:

if (cache_get(...) != NULL)
    return cache_got;

if ((result = fetch(...)) == NULL)
    return NULL;
cache_set(nid, properties, result)

tests are added or updated

levitte · 2019-02-13T11:23:48Z

I added this PR to the newly created github project "3.0 New Core + FIPS"

Also, you need to rebase.

levitte

Also, would you mind adding internal documentation for the functions declared in include/internal/property.h, i.e. a manpage in doc/internal/man3?

crypto/property/defn_cache.c

levitte · 2019-02-13T11:40:05Z

I removed the FIPS label, as this isn't the FIPS module per se. I believe adding this to the "3.0 New Core + FIPS" is a better way to mark these sorts of PRs

richsalz · 2019-02-13T14:01:54Z

crypto/property/defn_cache.c

+typedef struct {
+    const char *prop;
+    PROPERTY_LIST *defn;
+    char body[1];


"unwarranted chumminess with the compiler" (web-search it)
Pay the price of portable code and make this a pointer.

Double allocating and ruining cache coherency in code that's meant to be fast to avoid something that's always worked...

I think that's a misguided/premature optimization, at the expense of more clear code. We're going to be dispatching to crypto code :)

C99 allows using char body[]; Is C99 viable target for OpenSSL 3.0.0?

We can't use C99. However, this code is portable and safe. There are two restrictions: don't use an array of such structures (no space to use between records) and don't store a type other than char there (alignment).

richsalz · 2019-02-13T14:03:38Z

crypto/property/properties.ebnf

+          ( ',' ( '-'? PropertyName | PropertyName ( '=' | '!=' ) Value ) )*
+Value    ::= NumberLiteral
+        | StringLiteral
+StringLiteral ::= QuotedString | UnquotedString


I think unquotedstring is a very big mistake. It makes it really hard to add keywords to the query language in the future. Look at perl's "barewords" evolution, for example.

richsalz · 2019-02-13T14:03:56Z

crypto/property/properties.ebnf

+Value    ::= NumberLiteral
+        | StringLiteral
+StringLiteral ::= QuotedString | UnquotedString
+QuotedString     ::= '"' [^"]* '"'


I think you should allow backslash-escape inside the quoted string.

Why? so we can write fips="Y\e\s" ?

I'd be happy to ban \ inside a string to provide a capability to add it painlessly in the future.

Banning it for future expansion helps. But I meant to only escape quotes, as in things like "Hello "world""

If anyone is ever going to want to have a Windows path in a quoted string, we'd better not use \ as the escape character.

That being said, the \ is more or less universally known as an escape character, and is already used as such elsewhere in OpenSSL, so it's actually a little surprising to have something different for property values.

(and I'm curious, where was ^ inspired from? That's actually used in VMS file specs, where you'll see things like openssl-1^.0^.2q^.tar.gz, but I've not seen you as VMS savvy, so...)

where was ^ inspired from?

The caret at the beginning of a character class inverts the set, i.e. [^a-b] is the complement of [a-b]. If you want to include a literal caret, it must not be first in the set. See GREP(1).

(Of course you know that, it's part of your Perl-fu. It was just too early in the morning... ;-) ).

D'oh.

And still, that was after the first coffee.

I think the ^ is stretching eBNF a little.

crypto/property/README

crypto/property/defn_cache.c

include/openssl/lhash.h

test/property_test.c

mattcaswell · 2019-02-13T14:03:01Z

crypto/property/README

+  names and values to small integer indices.  Names and values are
+  stored in separate hash tables.  The two Boolean values "yes" and "no"
+  are populated as the first two members of the value table.  The string
+  "name" is automatically populated as the first property name, to enable


Where does this happen? I couldn't see it.

See ossl_property_parse_init in property_parse.c.

That function only seems to define "provider", "version", "fips" and "engine", but not "name"

The question is which names are we going to reserve?

I changed "name" to "provider" since it seemed more applicable but missed the README.

The fix is to not say which property names are pre-loaded, just to say that they are.

We can have the discussion about which get added later. I expect some worthwhile ones will naturally appear over the course of the project.

test/property_test.c

richsalz · 2019-02-13T14:04:31Z

crypto/property/property_string.c

+ * space and time efficient algorithms if this becomes a bottleneck.
+ */
+
+typedef struct {


unnamed struct isn't common. also same bad chumminess thing.

Unnamed struct might be uncommon, but naming it becomes a bit silly if that name is never used. I see no reason to add a struct name now.

crypto/property/defn_cache.c

include/internal/property.h

levitte · 2019-02-13T17:30:24Z

Looking through the code again and its intended use, I'm unsure that the functions declared in include/internal/property.h are named right.

That store (or those stores) are going to be used to save away methods, i.e. instances of EVP_CIPHER, EVP_MD, etc. Do we call them implementations or do we call them methods? Historically, they were called the latter (it's even more visible if you consider EVP_PKEY_METHOD), so I wonder of ossl_method_ isn't a better function prefix than ossl_impl_.

Food for thought.

include/internal/property.h

test/build.info

levitte · 2019-02-13T18:56:20Z

crypto/property/property.c

+
+    ossl_impl_store_cleanup();
+    return 0;
+}


Please make sure that this is called automatically, either as part of the init process, or in here with the help of a RUN_ONCE function (see #8225 to see how I do that)

levitte · 2019-02-13T18:56:55Z

crypto/property/property.c

+    ossl_prop_defn_cleanup();
+    ossl_impl_store_free(g_implementations);
+    g_implementations = NULL;
+}


Please make sure that this is called automatically, either as part of the cleanup process, or in here with the help of OPENSSL_atexit() (see #8225 to see how I do that)

crypto/property/property.c

paulidale · 2019-02-14T01:52:55Z

I've addressed most of the comments (resolving those that I did fix). ~~This mostly leaves the library context (& g_implementations) some parsing questions and~~ the run once.

paulidale · 2019-02-14T02:26:03Z

Looking through the code again and its intended use, I'm unsure that the functions declared in include/internal/property.h are named right.

~~Let's settle this before I do the documentation.~~

Settled.

paulidale · 2019-02-14T02:28:38Z

~~Travis failures look to be overly aggressive detection of unused macro generated functions. Do we have a standard way of prevent these without exposing the structures via a header file?~~

Fixed in a different PR.

crypto/property/README

paulidale · 2019-02-16T21:44:09Z

Tests are updated so that they don't reinitialise the properties anymore.
I think that's everything once the CIs pass.

paulidale · 2019-02-16T22:09:24Z

The CIs pass :)

t-j-h

I still don't think we should have bare PROPERTY and it should have an OSSL or OPENSSL prefix.

paulidale · 2019-02-16T22:51:51Z

Which PROPERTY ? Everything exposed outside of the crypto/property directory starts OSSL_ doesn't it? All functions exposed from crpyto/property also start ossl_. Some of the types don't, are they realistically a problem?

t-j-h · 2019-02-16T23:19:36Z

You have ossl_ prefixes on the functions expected to be used outside of the property directory but no prefix on the typedefs. They should be OSSL_ ... basically we don't want to run into clashes with other code and PS_IDX and PROPERTY are likely to clash.

Yes I know these aren't currently expected to be visible elsewhere - but they may evolve that way - and we should be clean in our "namespace" usage IMHO.

paulidale · 2019-02-16T23:41:56Z

I've prefixed the internal symbols.

t-j-h

Once the CIs show clear for the most recent change - approved.

paulidale · 2019-02-17T06:38:49Z

@levitte @mattcaswell I'll merge this in the morning (my time) unless there are objections.

paulidale · 2019-02-17T09:23:54Z

Squashed...

levitte · 2019-02-17T17:16:52Z

I'd like to run a test tomorrow against things I have that build on this...

levitte · 2019-02-17T17:17:24Z

(It looked good visually, though...)

paulidale · 2019-02-17T21:49:56Z

Let me know when you're done and I'll merge it or merge it yourself if the time-zones are immiscible.

Properties are a sequence of comma separated name=value pairs. A name without a corresponding value is assumed to be a Boolean and have the true value 'yes'. Values are either strings or numbers. Strings can be quoted either _"_ or _'_ or unquoted (with restrictions). There are no escape characters inside strings. Number are either decimal digits or '0x' followed by hexidecimal digits. Numbers are represented internally as signed sixty four bit values. Queries on properties are a sequence comma separated conditional tests. These take the form of name=value (equality test), name!=value (inequality test) or name (Boolean test for truth). Queries can be parsed, compared against a definition or merged pairwise.

Properties are a sequence of comma separated name=value pairs. A name without a corresponding value is assumed to be a Boolean and have the true value 'yes'. Values are either strings or numbers. Strings can be quoted either _"_ or _'_ or unquoted (with restrictions). There are no escape characters inside strings. Number are either decimal digits or '0x' followed by hexidecimal digits. Numbers are represented internally as signed sixty four bit values. Queries on properties are a sequence comma separated conditional tests. These take the form of name=value (equality test), name!=value (inequality test) or name (Boolean test for truth). Queries can be parsed, compared against a definition or merged pairwise. Reviewed-by: Matt Caswell <matt@openssl.org> Reviewed-by: Tim Hudson <tjh@openssl.org> (Merged from #8224)

Reviewed-by: Matt Caswell <matt@openssl.org> Reviewed-by: Tim Hudson <tjh@openssl.org> (Merged from #8224)

paulidale · 2019-02-18T06:35:24Z

Merged, thanks for the extensive and positive feedback.

levitte · 2019-02-18T07:59:49Z

... someone's in a rush. 😉

My tests ran well after some appropriate mods. The manual bits need some fixups, something I discovered moments ago...

paulidale · 2019-02-18T08:53:14Z

In a rush or needing a rush? I waited until I thought I’d be asleep (which I have been). That I’m not indicative is indactive of a non-problem…. Pauli -- Dr Paul Dale | Cryptographer | Network Security & Encryption Phone +61 7 3031 7217 Oracle Australia

…

On 18 Feb 2019, at 6:00 pm, Richard Levitte ***@***.***> wrote: ... someone's in a rush. 😉 My tests ran well after some appropriate mods. The manual bits need some fixups, something I discovered moments ago... — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#8224 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ARFiwdynDshahU7-Wch_GVzyCWHeZRDOks5vOl2IgaJpZM4a5LHK>.

levitte · 2019-02-18T09:05:27Z

Care to look at #8265?

paulidale added branch: master Merge to master branch FIPS labels Feb 13, 2019

levitte added this to In progress in 3.0 New Core + FIPS via automation Feb 13, 2019

levitte requested changes Feb 13, 2019

View reviewed changes

crypto/property/defn_cache.c Outdated Show resolved Hide resolved

3.0 New Core + FIPS automation moved this from In progress to Needs review Feb 13, 2019

levitte removed the FIPS label Feb 13, 2019

richsalz reviewed Feb 13, 2019

View reviewed changes

mattcaswell requested changes Feb 13, 2019

View reviewed changes

richsalz reviewed Feb 13, 2019

View reviewed changes

levitte requested changes Feb 13, 2019

View reviewed changes

include/internal/property.h Outdated Show resolved Hide resolved

levitte requested changes Feb 13, 2019

View reviewed changes

test/build.info Outdated Show resolved Hide resolved

levitte reviewed Feb 13, 2019

View reviewed changes

levitte requested changes Feb 13, 2019

View reviewed changes

crypto/property/property.c Show resolved Hide resolved

paulidale force-pushed the property branch 2 times, most recently from 82ed864 to 88c0401 Compare February 14, 2019 01:50

paulidale force-pushed the property branch from 88c0401 to 6e4b167 Compare February 14, 2019 02:12

paulidale changed the title ~~Property implementation~~ WIP: Property implementation Feb 14, 2019

paulidale force-pushed the property branch from 6e4b167 to a537c06 Compare February 14, 2019 02:43

slontis reviewed Feb 14, 2019

View reviewed changes

crypto/property/README Outdated Show resolved Hide resolved

slontis reviewed Feb 14, 2019

View reviewed changes

crypto/property/README Outdated Show resolved Hide resolved

paulidale force-pushed the property branch from 5e49014 to 544d17d Compare February 14, 2019 02:58

t-j-h reviewed Feb 16, 2019

View reviewed changes

t-j-h approved these changes Feb 16, 2019

View reviewed changes

t-j-h approved these changes Feb 17, 2019

View reviewed changes

paulidale force-pushed the property branch from bb4043a to 68fea5b Compare February 17, 2019 09:23

mattcaswell approved these changes Feb 17, 2019

View reviewed changes

mattcaswell added the approval: done This pull request has the required number of approvals label Feb 17, 2019

paulidale force-pushed the property branch from 68fea5b to 19d562b Compare February 17, 2019 21:08

paulidale force-pushed the property branch from 59d434f to 6207afc Compare February 18, 2019 03:28

paulidale added 2 commits February 18, 2019 13:28

generated files

6207afc

levitte pushed a commit that referenced this pull request Feb 18, 2019

generated files

3037d0a

Reviewed-by: Matt Caswell <matt@openssl.org> Reviewed-by: Tim Hudson <tjh@openssl.org> (Merged from #8224)

paulidale closed this Feb 18, 2019

3.0 New Core + FIPS automation moved this from Needs review to Done Feb 18, 2019

paulidale deleted the property branch February 18, 2019 06:35

levitte mentioned this pull request Feb 18, 2019

Property: naming and manual clarifiations #8265

Closed

Property implementation #8224

Property implementation #8224

Conversation

paulidale commented Feb 13, 2019 • edited by levitte

levitte commented Feb 13, 2019 • edited

levitte left a comment

Choose a reason for hiding this comment

levitte commented Feb 13, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paulidale Feb 14, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

levitte commented Feb 13, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paulidale commented Feb 14, 2019 • edited

paulidale commented Feb 14, 2019 • edited

paulidale commented Feb 14, 2019 • edited

paulidale commented Feb 16, 2019

paulidale commented Feb 16, 2019

t-j-h left a comment

Choose a reason for hiding this comment

paulidale commented Feb 16, 2019 • edited

t-j-h commented Feb 16, 2019

paulidale commented Feb 16, 2019

t-j-h left a comment

Choose a reason for hiding this comment

paulidale commented Feb 17, 2019

paulidale commented Feb 17, 2019

levitte commented Feb 17, 2019

levitte commented Feb 17, 2019

paulidale commented Feb 17, 2019

paulidale commented Feb 18, 2019

levitte commented Feb 18, 2019

paulidale commented Feb 18, 2019 via email

levitte commented Feb 18, 2019

paulidale commented Feb 13, 2019 •

edited by levitte

levitte commented Feb 13, 2019 •

edited

paulidale Feb 14, 2019 •

edited

levitte commented Feb 13, 2019 •

edited

paulidale commented Feb 14, 2019 •

edited

paulidale commented Feb 14, 2019 •

edited

paulidale commented Feb 14, 2019 •

edited

paulidale commented Feb 16, 2019 •

edited