ClientGenerator overhaul

[jskeet]

The code generator could do with an overhaul in various dimensions:

• We're still using Python 2, which is becoming an increasingly untenable position
• Our setup.py requires a version of django which is massively out of date and has security vulnerabilities (which may well not affect us)
• Even though our languages directory only has csharp, we still have all the other generator python files; let's prune that

In an ideal world, we can fix all of this with a "test" of "regenerate the world, and nothing should change".

I'll start having a look at it today.

[jskeet]

Currently a bunch of the tests fail when running on Windows. It would be nice to fix that as we go along too...

Log of tests failures:

Running api_exception_test.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
Running api_test.py
.........................
----------------------------------------------------------------------
Ran 25 tests in 0.563s

OK
Running configuration_test.py
...
----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK
Running cpp_generator_test.py
........
----------------------------------------------------------------------
Ran 8 tests in 0.007s

OK
Running cpp_import_manager_test.py
F
======================================================================
FAIL: testAddImportAndCommit (__main__.CppImportManagerTest)
testAddImportAndCommit (__main__.CppImportManagerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "cpp_import_manager_test.py", line 49, in testAddImportAndCommit
    self.assertFalse(self.import_manager.other_imports)
AssertionError: <googleapis.codegen.utilities.ordered_set.MutableOrderedSet object at 0x0000000003F3C1D0> is not false

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)
Running csharp_generator_test.py
.EEEEEEE
======================================================================
ERROR: testMemberNameConflictWithParentClass (__main__.MemberNameTest)
testMemberNameConflictWithParentClass (__main__.MemberNameTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "csharp_generator_test.py", line 263, in testMemberNameConflictWithParentClass
    gen.AnnotateApiForLanguage(gen.api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\api_library_generator.py", line 171, in AnnotateApiForLanguage
    self._AnnotateTree(the_api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\api_library_generator.py", line 186, in _AnnotateTree
    self.AnnotateApi(api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\csharp_generator.py", line 221, in AnnotateApi
    the_api.values['revision'])
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\csharp_generator.py", line 270, in _GetRevisionNumber
    raise ValueError("Cannot infer valid revision number")
ValueError: Cannot infer valid revision number

======================================================================
ERROR: testMemberNameConflictWithParentClassWithUnderscores (__main__.MemberNameTest)
testMemberNameConflictWithParentClassWithUnderscores (__main__.MemberNameTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "csharp_generator_test.py", line 285, in testMemberNameConflictWithParentClassWithUnderscores
    gen.AnnotateApiForLanguage(gen.api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\api_library_generator.py", line 171, in AnnotateApiForLanguage
    self._AnnotateTree(the_api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\api_library_generator.py", line 186, in _AnnotateTree
    self.AnnotateApi(api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\csharp_generator.py", line 221, in AnnotateApi
    the_api.values['revision'])
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\csharp_generator.py", line 270, in _GetRevisionNumber
    raise ValueError("Cannot infer valid revision number")
ValueError: Cannot infer valid revision number

======================================================================
ERROR: testAbsurdlyNestedButRealClassNames (__main__.NestedClassNameTest)
testAbsurdlyNestedButRealClassNames (__main__.NestedClassNameTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "csharp_generator_test.py", line 182, in testAbsurdlyNestedButRealClassNames
    gen.AnnotateApiForLanguage(gen.api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\api_library_generator.py", line 171, in AnnotateApiForLanguage
    self._AnnotateTree(the_api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\api_library_generator.py", line 186, in _AnnotateTree
    self.AnnotateApi(api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\csharp_generator.py", line 221, in AnnotateApi
    the_api.values['revision'])
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\csharp_generator.py", line 270, in _GetRevisionNumber
    raise ValueError("Cannot infer valid revision number")
ValueError: Cannot infer valid revision number

======================================================================
ERROR: testNestedArray (__main__.NestedClassNameTest)
testNestedArray (__main__.NestedClassNameTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "csharp_generator_test.py", line 194, in testNestedArray
    gen.AnnotateApiForLanguage(gen.api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\api_library_generator.py", line 171, in AnnotateApiForLanguage
    self._AnnotateTree(the_api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\api_library_generator.py", line 186, in _AnnotateTree
    self.AnnotateApi(api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\csharp_generator.py", line 221, in AnnotateApi
    the_api.values['revision'])
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\csharp_generator.py", line 270, in _GetRevisionNumber
    raise ValueError("Cannot infer valid revision number")
ValueError: Cannot infer valid revision number

======================================================================
ERROR: testNestedClassName (__main__.NestedClassNameTest)
testNestedClassName (__main__.NestedClassNameTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "csharp_generator_test.py", line 161, in testNestedClassName
    gen.AnnotateApiForLanguage(gen.api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\api_library_generator.py", line 171, in AnnotateApiForLanguage
    self._AnnotateTree(the_api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\api_library_generator.py", line 186, in _AnnotateTree
    self.AnnotateApi(api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\csharp_generator.py", line 221, in AnnotateApi
    the_api.values['revision'])
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\csharp_generator.py", line 270, in _GetRevisionNumber
    raise ValueError("Cannot infer valid revision number")
ValueError: Cannot infer valid revision number

======================================================================
ERROR: testPackagePath (__main__.PackagePathTest)
testPackagePath (__main__.PackagePathTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "csharp_generator_test.py", line 222, in testPackagePath
    gen.AnnotateApiForLanguage(gen.api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\api_library_generator.py", line 171, in AnnotateApiForLanguage
    self._AnnotateTree(the_api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\api_library_generator.py", line 186, in _AnnotateTree
    self.AnnotateApi(api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\csharp_generator.py", line 221, in AnnotateApi
    the_api.values['revision'])
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\csharp_generator.py", line 270, in _GetRevisionNumber
    raise ValueError("Cannot infer valid revision number")
ValueError: Cannot infer valid revision number

======================================================================
ERROR: testVersionWithDash (__main__.VersionWithDashTest)
testVersionWithDash (__main__.VersionWithDashTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "csharp_generator_test.py", line 238, in testVersionWithDash
    gen.AnnotateApiForLanguage(gen.api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\api_library_generator.py", line 171, in AnnotateApiForLanguage
    self._AnnotateTree(the_api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\api_library_generator.py", line 186, in _AnnotateTree
    self.AnnotateApi(api)
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\csharp_generator.py", line 221, in AnnotateApi
    the_api.values['revision'])
  File "C:\Users\skeet\Test\Projects\google-api-dotnet-client-golden\ClientGenerator\src\googleapis\codegen\csharp_generator.py", line 270, in _GetRevisionNumber
    raise ValueError("Cannot infer valid revision number")
ValueError: Cannot infer valid revision number

----------------------------------------------------------------------
Ran 8 tests in 0.008s

FAILED (errors=7)
Running dart_generator_test.py
.
----------------------------------------------------------------------
Ran 1 test in 0.002s

OK
Running data_types_test.py
..
----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK
Running data_value_test.py
..
----------------------------------------------------------------------
Ran 2 tests in 0.001s

OK
Running documenting_language_model_test.py
.
----------------------------------------------------------------------
Ran 1 test in 0.001s

OK
Running generate_library_test.py
...
----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK
Running generator_lookup_test.py
....
----------------------------------------------------------------------
Ran 4 tests in 0.000s

OK
Running generator_test.py
unexpected file: pp/
.unexpected file: app.yaml
unexpected file: pp/
unexpected file: pp/xxx
.unexpected file: app.yaml
unexpected file: pp/
unexpected file: pp/xxx
Funexpected file: app.yaml
unexpected file: pp/
unexpected file: pp/xxx
.
======================================================================
FAIL: testWalkTemplateTreeWithFilteredResult (__main__.GeneratorTest)
testWalkTemplateTreeWithFilteredResult (__main__.GeneratorTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "generator_test.py", line 93, in testWalkTemplateTreeWithFilteredResult
    self.VerifyPackageContains(['foo', 'bar'], must_not_contain=['pp/xxx'])
  File "generator_test.py", line 63, in VerifyPackageContains
    self.fail('Found unexpected file %s in archive' % path)
AssertionError: Found unexpected file pp/xxx in archive

----------------------------------------------------------------------
Ran 4 tests in 0.015s

FAILED (failures=1)
Running gwt_generator_test.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
Running java_generator_test.py
.....com.google.api.client.json.JsonString
com.google.api.client.util.DateTime
com.google.common.primitives.UnsignedLong
....FF
======================================================================
FAIL: testDefaultPath (__main__.JavaLanguageModelTest)
testDefaultPath (__main__.JavaLanguageModelTest)
Test the package path generation.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "java_generator_test.py", line 170, in testDefaultPath
    self.assertEquals('com/google/api/services/fake', gen.api.module.path)
AssertionError: 'com/google/api/services/fake' != 'com/google/fake'

======================================================================
FAIL: testDefaultPathWithPackagePathAndCanonicalName (__main__.JavaLanguageModelTest)
testDefaultPathWithPackagePathAndCanonicalName (__main__.JavaLanguageModelTest)
Test the package path generation.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "java_generator_test.py", line 199, in testDefaultPathWithPackagePathAndCanonicalName
    gen.api.module.path)
AssertionError: 'com/google/api/services/my/path/canonicalname' != 'com/google/my/path/canonicalname'

----------------------------------------------------------------------
Ran 11 tests in 0.004s

FAILED (failures=2)
Running java_import_manager_test.py
...
----------------------------------------------------------------------
Ran 3 tests in 0.001s

OK
Running language_model_test.py
...........
----------------------------------------------------------------------
Ran 11 tests in 0.003s

OK
Running php_generator_test.py
.....
----------------------------------------------------------------------
Ran 5 tests in 0.003s

OK
Running python_generator_test.py
...
----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK
Running schema_test.py
.......WARNING:root:object without properties NoProperties: {'type': 'object', 'id': 'NoProperties'}
...
----------------------------------------------------------------------
Ran 10 tests in 0.008s

OK
Running targets_test.py
.......
----------------------------------------------------------------------
Ran 7 tests in 0.003s

OK
Running template_helpers_test.py
..F............................
======================================================================
FAIL: testCache (__main__.TemplateHelpersTest)
testCache (__main__.TemplateHelpersTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "template_helpers_test.py", line 591, in testCache
    self.assertFalse(test_path in loader._cache)
AssertionError: True is not false

----------------------------------------------------------------------
Ran 31 tests in 0.023s

FAILED (failures=1)
Running template_objects_test.py
........
----------------------------------------------------------------------
Ran 8 tests in 0.002s

OK
Running unicode_test.py
.
----------------------------------------------------------------------
Ran 1 test in 0.005s

OK

Failures from that:

• cpp_import_manager_test.py
• csharp_generator_test.py (ouch!)
• generator_lookup_test.py
• java_generator_test.py
• template_helpers_test.py

[jskeet]

Hmm. Updating Django is proving non-trivial due to google/apis-client-generator#13 - we're using an undocumented function in Django which was removed in 1.9. I'm having a look to see whether we really need it...

[jskeet]

Okay, yes, it looks like we would need it. We might be able to refactor to use TemplateEngine which might support this use case, but it's hard to know when I have so little experience of Django.

[jskeet]

Closing this, as we're not expecting to do anything in the short term. We will have discussions about the long term future for REST client generation separately.