Fix test_policy_to_permissions test failing when there's no internet #158

pbaughman · 2019-09-13T22:52:21Z

This PR does the following:

Makes sros2 install an XML catalog to the 'share' directory with a copy of http://www.w3.org/2001/03/xml.xsd
~~Makes test_policy_to_permissions.py use the installed catalog so the test doesn't die when you can't reach www.w3.org~~
Makes sros2 modify the XML_CATALOG_FILES environment variable when it's imported so we use the cached schema instead of going out to the internet

A quick recap

You can observe the original problem by running colcon test on the sros2 package without an internet connection. The test will fail with

lxml.etree.XMLSchemaParseError: attribute use (unknown), attribute 'ref': The QName value '{http://www.w3.org/XML/1998/namespace}base' does not resolve to a(n) attribute declaration., line 34
- generated xml file: /path_to_workspace/build/sros2/pytest.xml -

After this PR, the test will succeed, even without an internet connection

Open Questions

I have only fixed the test. The only way I could figure out to get lxml to use the new catalog was to add the path to the XML_CATALOG_FILES environment variable. I put some extra start-up code in test_policy_to_permissions to append (or create) this environment variable.

~~Presumably, we would want to same behavior to happen if you were just using the sros2 package. I would like to give the package maintainers an opportunity to suggest how to do that.~~

~~If we want to keep the test-only fix, we should make ament_index_python a test_depend instead of an exec_depend~~

In the latest version of this PR, you get the caching at run time, not just test time

Reference:

The instructions I (more or less) followed are here: https://www.w3.org/blog/2008/09/caching-xml-data-at-install-ti/

Fixes test_policy_to_permissions failing #157

sros2/package.xml

sros2/test/test_policy_to_permissions.py

sros2/xml_cache/xml.xsd

ruffsl · 2019-09-13T23:07:52Z

sros2/test/test_policy_to_permissions.py

+        'xhtml-cache.xml'
+    )
+elif 'xml_cache/xhtml-cache.xml' not in os.environ['XML_CATALOG_FILES']:
+    os.environ['XML_CATALOG_FILES'] += ' ' + os.path.join(


It's not clear to me what delimiter lxml supports, be it a space or : or if lists of paths are supported at all. Do you know of an example somewhere? I only see it referenced here: http://xmlsoft.org/catalog.html

Oh, from here: https://stackoverflow.com/a/8391738/2577586

Note that entries in XML_CATALOG_FILES are space-separated URLs. You can use Python's pathname2url and urljoin (with file:) to generate the URL from a pathname.

Do we need to prefix the path with file://?

I don't know. I tested it manually by setting my XML_CATALOG_FILES environment variable to /etc/xml/catalog and verified that the test still passes when it appends to the existing environment variable.

On this xmlsoft.org page it doesn't mention anything about file://

The user can change the default catalog behaviour by redirecting queries to its own set of catalogs, this can be done by setting the XML_CATALOG_FILES environment variable to a list of catalogs, an empty one should deactivate loading the default /etc/xml/catalog default catalog

@ruffsl It seems to work OK. . . I can pass the test without an internet connection, and my final XML_CATALOG_FILES environment variable is:

/etc/xml/catalog file%3A///sros2/install/sros2/share/sros2/xml_cache/xhtml-cache.xml

sros2/xml_cache/xml.xsd

pbaughman · 2019-09-13T23:13:59Z

If take the following out of the test. . .

import os

import ament_index_python


if 'XML_CATALOG_FILES' not in os.environ:
    os.environ['XML_CATALOG_FILES'] = os.path.join(
        ament_index_python.get_package_share_directory('sros2'),
        'xml_cache',
        'xhtml-cache.xml'
    )
elif 'xml_cache/xhtml-cache.xml' not in os.environ['XML_CATALOG_FILES']:
    os.environ['XML_CATALOG_FILES'] += ' ' + os.path.join(
        ament_index_python.get_package_share_directory('sros2'),
        'xml_cache',
        'xhtml-cache.xml'
    )

. . and add it to the (currently empty) sros2/__init__.py file, that also solves the problem in a way that isn't test-only. Like I mentioned above, that seems hacky to me but I come from a bit of a different background and have been known to be to conservative about this kind of thing. If the package owners/maintainers are OK with it, I'll make the change

pbaughman · 2019-09-13T23:45:38Z

Ok, I put the code that extends the XML_CATALOG_FILES environment variable into __init__.py so it happens whenever you import sros2.

I also changed the path into a URL with 'file://' and the colon character replaced with %3A It seems to still work

pbaughman · 2019-09-16T23:14:24Z

@ruffsl Is there anybody else we should reach out to for feedback on this? Maybe @kyrofa who commented on the original issue, too? I'm not so familiar with everybody who works on this particular package.

kyrofa · 2019-09-16T23:24:43Z

Maybe @kyrofa who commented on the original issue, too?

I'm sorry to say that I'm not very familiar with this part of the package. Perhaps @mikaelarguedas has more context for how this fits together.

pbaughman · 2019-09-18T15:46:07Z

Can anybody suggest what I need to do to move this PR forward?

ruffsl · 2019-09-18T16:17:49Z

sros2/xml_cache/xhtml-cache.xml

@@ -0,0 +1,5 @@
+<catalog xmlns="urn:oasis:names:tc:entity:xmlns:xml:catalog">
+  <rewriteURI
+      uriStartString="http://www.w3.org/2001/03/"


Are there any other w3 documents that we use that would also have there uri prefix rewritten? Was the XML.xsd the only source of outbound network traffic?

The manual test I ran for this was:

Build and test 'master' and observe that the tests pass

Rip the ethernet cable out of the back of my computer (or if you want to be less dramatic sudo ifconfig eno2 down) and observe that the tests fail

Build and test this branch and observe that the tests pass again

Plug the ethernet cable back into my computer and observe that the tests still pass

So I'm pretty confident that this was the only external resource, unless there are other schemas that aren't tested as part of the unit test

If another external resource with the above prefix is used, but a local copy is not added to the XML cache, then the lookup will fail.

I suppose it's possible for some other python program to import sros2 (thus getting its XML catalog injected into its environment), then attempt a lookup of another document with a prefix http://www.w3.org/2001/03/something_else In that case, the lookup will fail.

I will change the prefix to

<rewriteURI uriStartString="http://www.w3.org/2001/03/xml.xsd" rewritePrefix="./xml.xsd" />

So that ONLY this file is cataloged. That should prevent us from breaking other XML lookups in the same python process as sros2

ruffsl · 2019-09-18T16:24:36Z

It might be handy to have a test that can assert the entire CLI works as expected when offline. Perhaps the test could force the python interpreter to use a non-existent proxy via temporary environment variable, then continue with the test suite, then unset it once finished? I'm not sure if that's possible.

I'll try and make some time this week to test the pr locally.

pbaughman · 2019-09-18T16:50:36Z

It might be handy to have a test that can assert the entire CLI works as expected when offline. Perhaps the test could force the python interpreter to use a non-existent proxy via temporary environment variable, then continue with the test suite, then unset it once finished? I'm not sure if that's possible.

I will investigate this, but a solution that works across platforms seems tricky

pbaughman · 2019-09-18T17:22:23Z

grepping around in the libxml2 sourcecode, I did find some promising environment variables:

pete.baughman@ade:~/libxml2 (master u=)$ grep -R 'getenv'
# . . .
nanohttp.c:     env = getenv("no_proxy");
nanohttp.c:     env = getenv("http_proxy");
nanohttp.c:     env = getenv("HTTP_PROXY");

Your suggestion may be easier to implement than I thought. I can set HTTP_PROXY=http://example.com and 'master' will fail even with an internet connection, while my fix branch passes.

pbaughman · 2019-09-18T20:41:09Z

@ruffsl Ok, I've added the requested test. There's also a sanity check test that's skipped because it takes around 60 seconds to pass. I don't mind a test that takes a long time to fail, but a test that takes a long time to pass is problematic. I'll let you decide what to do with it.

This test will fail on master and pass on this branch

pbaughman · 2019-09-20T15:06:56Z

@ruffsl Have you been able to reproduce the failure with the steps I gave? You can also break libxml2's ability to fetch a schema remotely by doing

export HTTP_PROXY=http://example.com

before running the test. That will also let you observe the failure on 'master' and see that things down fail using this branch

ruffsl

Added some suggestions. You'll want to ping @jacobperron or @mjcarroll from OSRF and ask to trigger a CI job when your ready.

ruffsl · 2019-09-20T17:00:22Z

sros2/test/sros2/test_cli.py

+    env['HTTP_PROXY'] = 'http://example.com'
+    tmp_dir = tempfile.mkdtemp(prefix='sros2_test')
+
+    subprocess.run(


Should the api be used directly, to ensure we're testing the expected library, rather whatever subprocess finds on the path? I think the other system tests use this pattern:

sros2/sros2/test/sros2/commands/security/verbs/test_create_key.py

Line 31 in 3689472

from sros2.api import create_keystore

@ruffsl I'm sorry, but I can't answer this question for you. - I added this test based on your statement
It might be handy to have a test that can assert the entire CLI works as expected when offline

I can try to find a way to make sure that subprocess finds the right executable, but you're on both sides of the question "Should we invoke the API, or should we use the CLI"

I think you should be calling the api, as this unit test is to check if the XML expansion works offline, not that the command line verb doesn't return a non-zero for some other reason.

@ruffsl If your goal is to reduce the scope of the test to check that XML expansion works offline, would you accept running just test_policy_to_permissions with the HTTP_PROXY environment set?

ruffsl · 2019-09-20T17:11:53Z

sros2/test/sros2/test_cli.py

+        args=['ros2', 'security', 'create_keystore', 'demo_keys'],
+        check=True,
+        cwd=tmp_dir,
+        env=env


If switching to calling the command via the api, perhaps you could use the same env_backup pattern you have bellow to set/restore the env before/after setup and teardown.

ruffsl · 2019-09-20T17:59:04Z

With respect to #158 (comment) , perhaps testing via added unit tests could be reapproached?

My earlier statement was thinking of more of adding the fake proxy env to the entire test environment, so that the test suite by default acts as if its offline as far as the lxml is concerned. That way, if any more online requirements sneak their way in the future, via added schema references or what have you, they'd always surface using existing unit tests.

I guess I'm saying that blaketing the entire test suite for sros2's cli to run offline would be simpler than adding separate offline-unit tests for every new verb or feature. I think the normative case is that sros2 cli/api should work entirely devoid of the internet, and if later we do add a feature that needs to connect to a remote resource, then we can unset the env for just those respective unit tests.

pbaughman · 2019-09-20T18:08:34Z

@ruffsl Ok, I will put in some pytest hooks to set up the environment for all sros2 unit tests

In the meantime, I am also encountering new problems because sros2 does not correctly install its package.xml, or register itself correctly with ament_index.

See colcon/colcon-ros#74 and https://discourse.ros.org/t/can-we-chat-about-breaking-changes-in-stable-releases/10689/16 for some background information

Also see the recent PR ros2/launch#323 for the fix to the same problem from the package 'launch'

I will open another PR to address the missing package marker and manifest, but right now this PR is blocked until that issue can be resolved

pbaughman · 2019-09-20T19:06:00Z

@ruffsl I have removed the test that used subprocess to run the CLI and have replaced it with a conftest.py file that disables lxml2 internet lookup for all unit tests.

Note that this branch also contains the changes from PR#160, so I'll need to rebase once that is merged. Without the changes from the fix_missing_ament_index_resource running colcon test on this branch will fail with

==================================== ERRORS ====================================
________________________ ERROR collecting test session _________________________
/opt/ros/eloquent/lib/python3.6/site-packages/ament_index_python/packages.py:49: in get_package_prefix
    content, package_prefix = get_resource('packages', package_name)
/opt/ros/eloquent/lib/python3.6/site-packages/ament_index_python/resources.py:49: in get_resource
    "Could not find the resource '%s' of type '%s'" % (resource_name, resource_type))
E   LookupError: Could not find the resource 'sros2' of type 'packages'

During handling of the above exception, another exception occurred:
/usr/local/lib/python3.6/dist-packages/pluggy/hooks.py:286: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
/usr/local/lib/python3.6/dist-packages/pluggy/manager.py:92: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
/usr/local/lib/python3.6/dist-packages/pluggy/manager.py:86: in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
/usr/local/lib/python3.6/dist-packages/_pytest/python.py:185: in pytest_collect_file
    return ihook.pytest_pycollect_makemodule(path=path, parent=parent)
/usr/local/lib/python3.6/dist-packages/pluggy/hooks.py:286: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
/usr/local/lib/python3.6/dist-packages/pluggy/manager.py:92: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
/usr/local/lib/python3.6/dist-packages/pluggy/manager.py:86: in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
/opt/ros/eloquent/lib/python3.6/site-packages/launch_testing/pytest/hooks.py:98: in pytest_pycollect_makemodule
    entrypoint = find_launch_test_entrypoint(path)
/opt/ros/eloquent/lib/python3.6/site-packages/launch_testing/pytest/hooks.py:92: in find_launch_test_entrypoint
    return getattr(path.pyimport(), 'generate_test_description', None)
/usr/local/lib/python3.6/dist-packages/py/_path/local.py:701: in pyimport
    __import__(modname)
sros2/__init__.py:26: in <module>
    ament_index_python.get_package_share_directory('sros2'),
/opt/ros/eloquent/lib/python3.6/site-packages/ament_index_python/packages.py:69: in get_package_share_directory
    return os.path.join(get_package_prefix(package_name), 'share', package_name)
/opt/ros/eloquent/lib/python3.6/site-packages/ament_index_python/packages.py:52: in get_package_prefix
    "package '{}' not found, searching: {}".format(package_name, get_search_paths()))
E   ament_index_python.packages.PackageNotFoundError: "package 'sros2' not found, searching: ['/sros2/install/sros2', '/opt/ros/eloquent']"

If you want to double-check that the conftest.py file works as advertised, you just need to copy that one file to 'master' and run the tests. They should fail - but it will take a while because I think lxml2 tries for 60 seconds before it gives up every time it tries to fetch an external resource

Signed-off-by: Pete Baughman <pete.baughman@apex.ai>

- This changes a test-only fix into a always fix - Make the path into a URL instead of just a local file path Signed-off-by: Pete Baughman <pete.baughman@apex.ai>

Signed-off-by: Pete Baughman <pete.baughman@apex.ai>

ruffsl · 2019-09-20T19:23:07Z

sros2/test/conftest.py

+
+# Disable lxml2 lookup of resources on the internet by configuring it to use a proxy
+# that does not exist
+os.environ['HTTP_PROXY'] = 'http://example.com'


I don't know enough about conftest.py, but will pytest unset this after testing this package? It might be an issue if the original proxy settings, be it empty/unset or somethings else for the build farm, is not restored afterwards.

@ruffsl os.environ will only modify the environment for the current process, so this will only affect the environment for pytests in the sros2 package

I guess one other important piece of information is that pytest is invoked (at least) once per package, so all the other packages get their own pytest with a clean set of environment variables.

Ah, TDIL. Thanks!

ruffsl

LGTM. @jacobperron or @mjcarroll , could either of you trigger a CI job?

jacobperron · 2019-09-20T22:19:27Z

Linux
Linux-aarch64
macOS
Windows

pbaughman · 2019-09-20T23:06:52Z

Looks like windows doesn't like a trailing slash in the catalog path

ValueError: path 'share/sros2/xml_cache/' cannot end with '/'

I'll take another crack when I'm back at a computer w/ dev tools on it

- This should fix the windows CI job Signed-off-by: Pete Baughman <pete.baughman@apex.ai>

pbaughman · 2019-09-23T15:02:32Z

@jacobperron Can we try once more? Thanks

jacobperron · 2019-09-23T17:36:44Z

Windows

Edit: retriggered with up-to-date repos file.

pbaughman · 2019-09-23T18:21:01Z

Sigh. . . OK I'm going to figure out to get a windows machine to develop this feature. It doesn't appear that lxml is using the local cataloged schema document on Windows.

Signed-off-by: Pete Baughman <pete.baughman@apex.ai>

pbaughman · 2019-09-25T03:48:09Z

@ruffsl Windows doesn't like urllib.request.pathname2url I don't think Linux needs it but I'll double check tomorrow - I just took it from the stackoverflow page you linked. If Linux works without pathname2url, do you mind if I drop it?

Recap:

# Does not work on windows
_xml_cache_path = urllib.request.pathname2url(
    urllib.parse.urljoin( 
        'file:', 
        'C:/Users/user/sros2/sros2/xml_cache/xhtml-cache.xml'
    ) 
)

# Works on Windows
_xml_cache_path = urllib.parse.urljoin( 
    'file:', 
    'C:/Users/user/sros2/sros2/xml_cache/xhtml-cache.xml'
)

# Works on Windows (sanity check)
_xml_cache_path = 'C:/Users/user/sros2/sros2/xml_cache/xhtml-cache.xml'

# Does not work on windows (sanity check)
_xml_cache_path = ''

ruffsl · 2019-09-25T04:35:31Z

If Linux works without pathname2url, do you mind if I drop it?

Will that still be safe if/when the file path includes non-url safe characters, like spaces, etc?
Given from the same stackoverflow answer, XML_CATALOG_FILES is already "space-separated".

ruffsl · 2019-09-25T04:40:59Z

Related? vulnersCom/getsploit#1

pbaughman · 2019-09-25T14:40:29Z

Ok, I'll try putting pathname2url around just the path portion and not the 'file:' portion, or trying to quote the path instead.

Edit will also test with paths that contain spaces on windows and linux.

This works on linux with or without spaces in the path:

_xml_cache_path = urllib.parse.urljoin(
    'file:',
    urllib.request.pathname2url(
        os.path.join(
                ament_index_python.get_package_share_directory('sros2'),
                'xml_cache',
                'xhtml-cache.xml'
        )
    )
)

Will have to wait until I go home to test on Windows

This should work on linux AND on windows Signed-off-by: Pete Baughman <pete.baughman@apex.ai>

pbaughman · 2019-09-25T23:09:53Z

@jacobperron Let's try one more time. This works on my Ubuntu machine at work, and on my windows machine at home.

jacobperron · 2019-09-25T23:11:30Z

Linux
Linux-aarch64
macOS
Windows

…os2#158) * Install an XML catalog so we can look this schema up locally Signed-off-by: Pete Baughman <pete.baughman@apex.ai> * Mangle the XML_CATALOG_FILE when you import sros2 - This changes a test-only fix into a always fix - Make the path into a URL instead of just a local file path Signed-off-by: Pete Baughman <pete.baughman@apex.ai> * Disable lxml2 resource lookup over the internet for tests Signed-off-by: Pete Baughman <pete.baughman@apex.ai> * Remove trailing slash from path - This should fix the windows CI job Signed-off-by: Pete Baughman <pete.baughman@apex.ai> * Add README.md explaining where the cataloged schema comes from Signed-off-by: Pete Baughman <pete.baughman@apex.ai> * Escape the path, not the 'file:' part This should work on linux AND on windows Signed-off-by: Pete Baughman <pete.baughman@apex.ai>

pbaughman · 2019-09-27T18:39:14Z

Thanks so much guys! Any chance we can also get this on Dashing ? I've got a version here that has the three necessary commits to make this build and test successfully on the docker image ros:dashing

ruffsl · 2019-09-27T19:22:41Z

I think the most recent Dashing sync was set for 2019-09-24, but you could chime in on the topic to ask if the sync is already out the door, or if we could squeeze in this patch:
https://discourse.ros.org/t/preparing-for-dashing-sync-2019-09-24/10705

kyrofa · 2019-09-27T19:24:12Z

We probably need to get it backported to the dashing branch before we start that conversation, though. I suspect we're too late for that sync.

pbaughman · 2019-09-27T19:36:39Z

Just getting it onto the dashing branch is enough for us

ruffsl · 2019-09-27T19:43:36Z

Then you may need to open a rebased PR at dashing branch so @jacobperron can test CI and merge.

kyrofa · 2019-09-27T19:44:24Z

My only concern about getting it back into dashing is the additional exec_depend, but we can chat about that on the backport PR.

…os2#158) * Install an XML catalog so we can look this schema up locally Signed-off-by: Pete Baughman <pete.baughman@apex.ai> * Mangle the XML_CATALOG_FILE when you import sros2 - This changes a test-only fix into a always fix - Make the path into a URL instead of just a local file path Signed-off-by: Pete Baughman <pete.baughman@apex.ai> * Disable lxml2 resource lookup over the internet for tests Signed-off-by: Pete Baughman <pete.baughman@apex.ai> * Remove trailing slash from path - This should fix the windows CI job Signed-off-by: Pete Baughman <pete.baughman@apex.ai> * Add README.md explaining where the cataloged schema comes from Signed-off-by: Pete Baughman <pete.baughman@apex.ai> * Escape the path, not the 'file:' part This should work on linux AND on windows Signed-off-by: Pete Baughman <pete.baughman@apex.ai>

…158) (#161) * Install an XML catalog so we can look this schema up locally Signed-off-by: Pete Baughman <pete.baughman@apex.ai> * Mangle the XML_CATALOG_FILE when you import sros2 - This changes a test-only fix into a always fix - Make the path into a URL instead of just a local file path Signed-off-by: Pete Baughman <pete.baughman@apex.ai> * Disable lxml2 resource lookup over the internet for tests Signed-off-by: Pete Baughman <pete.baughman@apex.ai> * Remove trailing slash from path - This should fix the windows CI job Signed-off-by: Pete Baughman <pete.baughman@apex.ai> * Add README.md explaining where the cataloged schema comes from Signed-off-by: Pete Baughman <pete.baughman@apex.ai> * Escape the path, not the 'file:' part This should work on linux AND on windows Signed-off-by: Pete Baughman <pete.baughman@apex.ai>

pbaughman mentioned this pull request Sep 13, 2019

test_policy_to_permissions failing #157

Closed

pbaughman commented Sep 13, 2019

View reviewed changes

sros2/package.xml Show resolved Hide resolved

pbaughman commented Sep 13, 2019

View reviewed changes

sros2/test/test_policy_to_permissions.py Outdated Show resolved Hide resolved

pbaughman commented Sep 13, 2019

View reviewed changes

sros2/xml_cache/xml.xsd Show resolved Hide resolved

ruffsl reviewed Sep 13, 2019

View reviewed changes

ruffsl reviewed Sep 18, 2019

View reviewed changes

ruffsl reviewed Sep 20, 2019

View reviewed changes

pbaughman mentioned this pull request Sep 20, 2019

Fix missing resources needed for ament #160

Merged

pbaughman force-pushed the fix_issue_157 branch 2 times, most recently from dfe54d9 to dc6b2c0 Compare September 20, 2019 19:00

pbaughman added 3 commits September 20, 2019 12:19

Install an XML catalog so we can look this schema up locally

432c949

Signed-off-by: Pete Baughman <pete.baughman@apex.ai>

Mangle the XML_CATALOG_FILE when you import sros2

06af87e

- This changes a test-only fix into a always fix - Make the path into a URL instead of just a local file path Signed-off-by: Pete Baughman <pete.baughman@apex.ai>

Disable lxml2 resource lookup over the internet for tests

9a21010

Signed-off-by: Pete Baughman <pete.baughman@apex.ai>

pbaughman force-pushed the fix_issue_157 branch from dc6b2c0 to 9a21010 Compare September 20, 2019 19:20

ruffsl reviewed Sep 20, 2019

View reviewed changes

Remove trailing slash from path

ddc358d

- This should fix the windows CI job Signed-off-by: Pete Baughman <pete.baughman@apex.ai>

Add README.md explaining where the cataloged schema comes from

148c43a

Signed-off-by: Pete Baughman <pete.baughman@apex.ai>

Escape the path, not the 'file:' part

6e949b9

This should work on linux AND on windows Signed-off-by: Pete Baughman <pete.baughman@apex.ai>

pbaughman force-pushed the fix_issue_157 branch from 46a8e2e to 6e949b9 Compare September 25, 2019 23:07

jacobperron approved these changes Sep 26, 2019

View reviewed changes

jacobperron merged commit fc2d859 into ros2:master Sep 26, 2019

pbaughman deleted the fix_issue_157 branch September 27, 2019 16:46

pbaughman mentioned this pull request Sep 27, 2019

Install local cache of XML schemas on 'dashing' #161

Merged

Fix test_policy_to_permissions test failing when there's no internet #158

Fix test_policy_to_permissions test failing when there's no internet #158

Conversation

pbaughman commented Sep 13, 2019 • edited

A quick recap

Open Questions

Reference:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pbaughman commented Sep 13, 2019 • edited

pbaughman commented Sep 13, 2019

pbaughman commented Sep 16, 2019

kyrofa commented Sep 16, 2019

pbaughman commented Sep 18, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruffsl commented Sep 18, 2019

pbaughman commented Sep 18, 2019

pbaughman commented Sep 18, 2019

pbaughman commented Sep 18, 2019

pbaughman commented Sep 20, 2019

ruffsl left a comment

Choose a reason for hiding this comment

ruffsl Sep 20, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruffsl commented Sep 20, 2019 • edited

pbaughman commented Sep 20, 2019

pbaughman commented Sep 20, 2019

ruffsl Sep 20, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pbaughman Sep 20, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruffsl left a comment

Choose a reason for hiding this comment

jacobperron commented Sep 20, 2019

pbaughman commented Sep 20, 2019

pbaughman commented Sep 23, 2019

jacobperron commented Sep 23, 2019 • edited

pbaughman commented Sep 23, 2019

pbaughman commented Sep 25, 2019

ruffsl commented Sep 25, 2019

ruffsl commented Sep 25, 2019

pbaughman commented Sep 25, 2019 • edited

pbaughman commented Sep 25, 2019

jacobperron commented Sep 25, 2019

pbaughman commented Sep 27, 2019

ruffsl commented Sep 27, 2019

kyrofa commented Sep 27, 2019

pbaughman commented Sep 27, 2019

ruffsl commented Sep 27, 2019

kyrofa commented Sep 27, 2019

pbaughman commented Sep 13, 2019 •

edited

pbaughman commented Sep 13, 2019 •

edited

ruffsl Sep 20, 2019 •

edited

ruffsl commented Sep 20, 2019 •

edited

ruffsl Sep 20, 2019 •

edited

pbaughman Sep 20, 2019 •

edited

jacobperron commented Sep 23, 2019 •

edited

pbaughman commented Sep 25, 2019 •

edited