Skip to content


Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP


Updates Scala specification for 2.10 #20

wants to merge 1 commit into from

5 participants

soc Josh Suereth Paolo G. Giarrusso Adriaan Moors odersky
  • Removed Octal literals
  • Disallowed FP literals without digit after dot
  • Removed val in for comprehension value definitions (Fixes SI-4918)
  • Added ## to the list of non-NPE-throwing Null methods (Fixes SI-4311)
  • Adapted some wording around AnyRef/AnyVal
  • Removed primitive type aliases (Fixes SI-4628)
  • Casting of null (Fixes second part of SI-4437)
  • Removed ScalaObject from the spec (Closes SI-4648)
  • Merged the changes of the PDF-only Scala-2.9-Draft manually
soc soc Updates Scala specification for 2.10
- Removed Octal literals
- Disallowed FP literals without digit after dot
- Removed val in for comprehension value definitions (Fixes SI-4918)
- Added ## to the list of non-NPE-throwing Null methods (Fixes SI-4311)
- Adapted some wording around AnyRef/AnyVal
- Removed primitive type aliases (Fixes SI-4628)
- Casting of null (Fixes second part of SI-4437)
- Removed ScalaObject from the spec (Closes SI-4648)
- Merged the changes of the PDF-only Scala-2.9-Draft manually
Josh Suereth

Review by: @odersky


Any news on this?

  • Removed Octal literals is SI-5205
  • Disallowed FP literals without digit after dot is related to SI-265 (found no better issue)

I added links from the issues to this pull request, so that people don't mistakenly duplicate work.

Adriaan Moors adriaanm referenced this pull request in scala/scala

SI-7292 Deprecate octal escape literals #2342

Paolo G. Giarrusso Blaisorblade commented on the diff
((8 lines not shown))
-trait Dynamic {
- def applyDynamic (name: String, args: Any*): Any
- ...
-Assume a selection of the form $e.x$ where the type of $e$ conforms to \lstinline@scala.Dynamic@.
-Further assuming the selection is not followed by any function arguments, such an expression can be rewitten under the conditions given in \sref{sec:impl-conv} to:
-If the selection is followed by some arguments, e.g.\ $e.x(\args)$, then that expression
-is rewritten to
-$e$.applyDynamic("$x$", $\args$)

This doesn't seem mentioned in the issue description, and Scala does have scala.Dynamic (with some slightly different details). So I'm against merging this bit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Paolo G. Giarrusso

I'd hope that some technical uncontroversial changes (say, to the grammar) could be merged after review by @adriaanm. If that were possible (and I think that should be possible and overdue), probably splitting somewhat the changes of this commit would help reviewing. Personally I'd have a commit per fixed issue, but that's my taste.

OTOH, I can totally get why @odersky would restrict maintenance of, say, the type system spec to type theory experts (or even to himself).

@soc wrote:

Merged the changes of the PDF-only Scala-2.9-Draft manually

Is that why you removed so much from documentation/src/reference/ImplementationStatus.tex and documentation/src/reference/RationalePart.tex? Because such changes can only be reviewed by @odersky himself, unless he already did them.

So I'd say that should be a separate commit at least for this part, also to ease reviewing.

Also, why is so much removed from "Dynamic Member Selection"?

Paolo G. Giarrusso

OTOH, I can totally get why @odersky would restrict maintenance of, say, the type system spec to type theory experts (or even to himself).

But yet again OTOH, I'd say that changes like scala/scala@e28c3ed should also be reflected in the spec: I'd like to know what exactly that change does, and I wouldn't want to debug a type error following the affected part of the spec.


@Blaisorblade I don't think the spec is updated anymore in its current form. I guess this commit probably only exists to remind people of the sad state. Sadly, I have heard no news about the new spec format since the guy who did it was hired (by Google?!) ... not sure about the details anymore.

Paolo G. Giarrusso

Last I knew, Martin had exclusive maintainership of the spec and was swamped on other work, I thought that was the reason (can't find the message in scala-internals or Google unfortunately). And a (partial? abandoned?) migration to some new format sounds a less valid argument. Would it be possible to do something?


Well, there is not much I can do about it ... I have accepted that the turn-around time has to be measured in months (spec), weeks (code) or years (website).

Adriaan Moors

I have scheduled some time to update the spec, but due to limited resources it'll have to wait until M5 at the earliest.

soc commented

Nice to hear! (Although this pull request is probably not what you are looking for.) I had already split up the changes into 1 PR per change, but there hasn't been much progress recently (Iain seems to be pretty busy + more reasons here: iainmcgin/scala-ref-markdown#19):

George Leontiev folone referenced this pull request in scala/scala-lang

Update language reference link #80


Thanks for the changes. (and sorry for not responding sooner, I was not aware of the request). Can you please split the pull request into separate ones for each topic? Most of them look good but a few warrant further discussion. For instance, I am inclined to leave the rationale in for historic reasons. Many thanks in advance.

I'm currently extremely busy (exams), but I'll split them as soon as I have some time.


@odersky Done. See #97, #98, #99, #100, #101, #102, #103.

odersky odersky closed this
Cheng Lian liancheng referenced this pull request from a commit in liancheng/scala-dist
Michael Armbrust marmbrus SPARK-1251 Support for optimizing and executing structured queries
This pull request adds support to Spark for working with structured data using a simple SQL dialect, HiveQL and a Scala Query DSL.

*This is being contributed as a new __alpha component__ to Spark and does not modify Spark core or other components.*

The code is broken into three primary components:
 - Catalyst (sql/catalyst) - An implementation-agnostic framework for manipulating trees of relational operators and expressions.
 - Execution (sql/core) - A query planner / execution engine for translating Catalyst’s logical query plans into Spark RDDs.  This component also includes a new public interface, SqlContext, that allows users to execute SQL or structured scala queries against existing RDDs and Parquet files.
 - Hive Metastore Support (sql/hive) - An extension of SqlContext called HiveContext that allows users to write queries using a subset of HiveQL and access data from a Hive Metastore using Hive SerDes.  There are also wrappers that allows users to run queries that include Hive UDFs, UDAFs, and UDTFs.

A more complete design of this new component can be found in [the associated JIRA](

[An updated version of the Spark documentation, including API Docs for all three sub-components,]( is also available for review.

With this PR comes support for inferring the schema of existing RDDs that contain case classes.  Using this information, developers can now express structured queries that are automatically compiled into RDD operations.

// Define the schema using a case class.
case class Person(name: String, age: Int)
val people: RDD[Person] =
  sc.textFile("people.txt").map(_.split(",")).map(p => Person(p(0), p(1).toInt))

// The following is the same as 'SELECT name FROM people WHERE age >= 10 && age <= 19'
val teenagers = people.where('age >= 10).where('age <= 19).select('name).toRdd

RDDs can also be registered as Tables, allowing SQL queries to be written over them.
val teenagers = sql("SELECT name FROM people WHERE age >= 10 && age <= 19")

The results of queries are themselves RDDs and support standard RDD operations:
```scala => "Name: " + t(0)).collect().foreach(println)

Finally, with the optional Hive support, users can read and write data located in existing Apache Hive deployments using HiveQL.
sql("LOAD DATA LOCAL INPATH 'src/main/resources/kv1.txt' INTO TABLE src")

// Queries are expressed in HiveQL
sql("SELECT key, value FROM src").collect().foreach(println)

## Relationship to Shark

Unlike Shark, Spark SQL does not act as a drop in replacement for Hive or the HiveServer. Instead this new feature is intended to make it easier for Spark developers to run queries over structured data, using either SQL or the query DSL. After this sub-project graduates from Alpha status it will likely become a new optimizer/backend for the Shark project.

Author: Michael Armbrust <>
Author: Yin Huai <>
Author: Reynold Xin <>
Author: Lian, Cheng <>
Author: Andre Schumacher <>
Author: Yin Huai <>
Author: Timothy Chen <>
Author: Cheng Lian <>
Author: Timothy Chen <>
Author: Henry Cook <>
Author: Mark Hamstra <>

Closes #146 from marmbrus/catalyst and squashes the following commits:

458bd1b [Michael Armbrust] Update people.txt
0d638c3 [Michael Armbrust] Typo fix from @ash211.
bdab185 [Michael Armbrust] Address another round of comments: * Doc examples can now copy/paste into spark-shell. * SQLContext is serializable * Minor parser bugs fixed * Self-joins of RDDs now handled correctly. * Removed deprecated examples * Removed deprecated parquet docs * Made more of the API private * Copied all the DSLQuery tests and rewrote them as SQLQueryTests
778299a [Michael Armbrust] Fix some old links to
fead0b6 [Michael Armbrust] Create a new RDD type, SchemaRDD, that is now the return type for all SQL operations.  This improves the old API by reducing the number of implicits that are required, and avoids throwing away schema information when returning an RDD to the user.  This change also makes it slightly less verbose to run language integrated queries.
fee847b [Michael Armbrust] Merge remote-tracking branch 'origin/master' into catalyst, integrating changes to serialization for ShuffledRDD.
48a99bc [Michael Armbrust] Address first round of feedback.
461581c [Michael Armbrust] Blacklist test that depends on JVM specific rounding behaviour
adcf1a4 [Henry Cook] Update
9dffbfa [Michael Armbrust] Style fixes. Add downloading of test cases to jenkins.
6978dd8 [Michael Armbrust] update docs, add apache license
1d0eb63 [Michael Armbrust] update changes with spark core
e5e1d6b [Michael Armbrust] Remove travis configuration.
c2efad6 [Michael Armbrust] First draft of SQL documentation.
013f62a [Michael Armbrust] Fix documentation / code style.
c01470f [Michael Armbrust] Clean up example
2f22454 [Michael Armbrust] WIP: Parquet example.
ce8073b [Michael Armbrust] clean up implicits.
f7d992d [Michael Armbrust] Naming / spelling.
9eb0294 [Michael Armbrust] Bring expressions implicits into SqlContext.
d2d9678 [Michael Armbrust] Make sure hive isn't in the assembly jar.  Create a separate, optional Hive assembly that is used when present.
8b35e0a [Michael Armbrust] address feedback, work on DSL
5d71074 [Michael Armbrust] Merge pull request #62 from AndreSchumacher/parquet_file_fixes
f93aa39 [Andre Schumacher] Better handling of path names in ParquetRelation
1a4bbd9 [Michael Armbrust] Merge pull request #60 from marmbrus/maven
3386e4f [Michael Armbrust] Merge pull request #58 from AndreSchumacher/parquet_fixes
3447c3e [Michael Armbrust] Don't override the metastore / warehouse in non-local/test hive context.
7233a74 [Michael Armbrust] initial support for maven builds
f0ba39e [Michael Armbrust] Merge remote-tracking branch 'origin/master' into maven
7386a9f [Michael Armbrust] Initial example programs using spark sql.
aeaef54 [Andre Schumacher] Removing unnecessary Row copying and reverting some changes to MutableRow
7ca4b4e [Andre Schumacher] Improving checks in Parquet tests
5bacdc0 [Andre Schumacher] Moving towards mutable rows inside ParquetRowSupport
54637ec [Andre Schumacher] First part of second round of code review feedback
c2a658d [Michael Armbrust] Merge pull request #55 from marmbrus/mutableRows
ba28849 [Michael Armbrust] code review comments.
d994333 [Michael Armbrust] Remove copies before shuffle, this required changing the default shuffle serialization.
9049cf0 [Michael Armbrust] Extend MutablePair interface to support easy syntax for in-place updates.  Also add a constructor so that it can be serialized out-of-the-box.
959bdf0 [Michael Armbrust] Don't silently swallow all KryoExceptions, only the one that indicates the end of a stream.
d371393 [Michael Armbrust] Add a framework for dealing with mutable rows to reduce the number of object allocations that occur in the critical path.
c9f8fb3 [Michael Armbrust] Merge pull request #53 from AndreSchumacher/parquet_support
3c3f962 [Michael Armbrust] Fix a bug due to array reuse.  This will need to be revisited after we merge the mutable row PR.
7d0f13e [Michael Armbrust] Update parquet support with master.
9d419a6 [Michael Armbrust] Merge remote-tracking branch 'catalyst/catalystIntegration' into parquet_support
0040ae6 [Andre Schumacher] Feedback from code review
1ce01c7 [Michael Armbrust] Merge pull request #56 from liancheng/unapplySeqForRow
70e489d [Cheng Lian] Fixed a spelling typo
6d315bb [Cheng Lian] Added Row.unapplySeq to extract fields from a Row object.
8d5da5e [Michael Armbrust] modify to include datanucleus jars explicitly
99e61fb [Michael Armbrust] Merge pull request #51 from marmbrus/expressionEval
7b9d142 [Michael Armbrust] Update travis to increase permgen size.
da9afbd [Michael Armbrust] Add byte wrappers for hive UDFS.
6fdefe6 [Michael Armbrust] Port sbt improvements from master.
296fe50 [Michael Armbrust] Address review feedback.
d7fbc3a [Michael Armbrust] Several performance enhancements and simplifications of the expression evaluation framework.
3bda72d [Andre Schumacher] Adding license banner to new files
3ac9eb0 [Andre Schumacher] Rebasing to new main branch
c863bed [Andre Schumacher] Codestyle checks
61e3bfb [Andre Schumacher] Adding WriteToFile operator and rewriting ParquetQuerySuite
3321195 [Andre Schumacher] Fixing one import in ParquetQueryTests.scala
3a0a552 [Andre Schumacher] Reorganizing Parquet table operations
18fdc44 [Andre Schumacher] Reworking Parquet metadata in relation and adding CREATE TABLE AS for Parquet tables
75262ee [Andre Schumacher] Integrating operations on Parquet files into SharkStrategies
f347273 [Andre Schumacher] Adding ParquetMetaData extraction, fixing schema projection
6a6bf98 [Andre Schumacher] Added column projections to ParquetTableScan
0f17d7b [Andre Schumacher] Rewriting ParquetRelation tests with RowWriteSupport
a11e364 [Andre Schumacher] Adding Parquet RowWriteSupport
6ad05b3 [Andre Schumacher] Moving ParquetRelation to spark.sql core
eb0e521 [Andre Schumacher] Fixing package names and other problems that came up after the rebase
99a9209 [Andre Schumacher] Expanding ParquetQueryTests to cover all primitive types
b33e47e [Andre Schumacher] First commit of Parquet import of primitive column types
c334386 [Michael Armbrust] Initial support for generating schema's based on case classes.
608a29e [Michael Armbrust] Add hive as a repl dependency
7413ac2 [Michael Armbrust] make test downloading quieter.
4d57d0e [Michael Armbrust] Fix test execution on travis.
5f2963c [Michael Armbrust] naming and continuous compilation fixes.
f5e7492 [Michael Armbrust] Add Apache license.  Make naming more consistent.
3ac9416 [Michael Armbrust] Merge support for working with schema-ed RDDs using catalyst in as a spark subproject.
2225431 [Michael Armbrust] Merge pull request #48 from marmbrus/minorFixes
d393d2a [Michael Armbrust] Review Comments: Add comment to map that adds a sub query.
24eaa79 [Michael Armbrust] fix > 100 chars
6e04e5b [Michael Armbrust] Add insertIntoTable to the DSL.
df88f01 [Michael Armbrust] add a simple test for aggregation
18a861b [Michael Armbrust] Correctly convert nested products into nested rows when turning scala data into catalyst data.
b922511 [Michael Armbrust] Fix insertion of nested types into hive tables.
5fe7de4 [Michael Armbrust] Move table creation out of rule into a separate function.
a430895 [Michael Armbrust] Planning for logical Repartition operators.
532dd37 [Michael Armbrust] Allow the local warehouse path to be specified.
4905b2b [Michael Armbrust] Add more efficient TopK that avoids global sort for logical Sort => StopAfter.
8c01c24 [Michael Armbrust] Move definition of Row out of execution to top level sql package.
c9116a6 [Michael Armbrust] Add combiner to avoid NPE when spark performs external aggregation.
29effad [Michael Armbrust] Include alias in attributes that are produced by overridden tables.
9990ec7 [Michael Armbrust] Merge pull request #28 from liancheng/columnPruning
f22df3a [Michael Armbrust] Merge pull request #37 from yhuai/SerDe
cf4db59 [Lian, Cheng] Added golden answers for PruningSuite
54f165b [Lian, Cheng] Fixed spelling typo in two golden answer file names
2682f72 [Lian, Cheng] Merge remote-tracking branch 'origin/master' into columnPruning
c5a4fab [Lian, Cheng] Merge branch 'master' into columnPruning
f670c8c [Yin Huai] Throw a NotImplementedError for not supported clauses in a CTAS query.
128a9f8 [Yin Huai] Minor changes.
017872c [Yin Huai] Remove stats20 from whitelist.
a1a4776 [Yin Huai] Update comments.
feb022c [Yin Huai] Partitioning key should be case insensitive.
555fb1d [Yin Huai] Correctly set the extension for a text file.
d00260b [Yin Huai] Strips backticks from partition keys.
334aace [Yin Huai] New golden files.
a40d6d6 [Yin Huai] Loading the static partition specified in a INSERT INTO/OVERWRITE query.
428aff5 [Yin Huai] Distinguish `INSERT INTO` and `INSERT OVERWRITE`.
eea75c5 [Yin Huai] Correctly set codec.
45ffb86 [Yin Huai] Merge remote-tracking branch 'upstream/master' into SerDeNew
e089627 [Yin Huai] Code style.
563bb22 [Yin Huai] Set compression info in FileSinkDesc.
35c9a8a [Michael Armbrust] Merge pull request #46 from marmbrus/reviewFeedback
bdab5ed [Yin Huai] Add a TODO for loading data into partitioned tables.
5495fab [Yin Huai] Remove cloneRecords which is no longer needed.
1596e1b [Yin Huai] Cleanup imports to make IntelliJ happy.
3bb272d [Michael Armbrust] move org.apache.spark.sql package.scala to the correct location.
8506c17 [Michael Armbrust] Address review feedback.
3cb4f2e [Michael Armbrust] Merge pull request #45 from tnachen/master
9ad474d [Michael Armbrust] Merge pull request #44 from marmbrus/sampling
566fd66 [Timothy Chen] Whitelist tests and add support for Binary type
69adf72 [Yin Huai] Set cloneRecords to false.
a9c3188 [Timothy Chen] Fix udaf struct return
346f828 [Yin Huai] Move SharkHadoopWriter to the correct location.
59e37a3 [Yin Huai] Merge remote-tracking branch 'upstream/master' into SerDeNew
ed3a1d1 [Yin Huai] Load data directly into Hive.
7f206b5 [Michael Armbrust] Add support for hive TABLESAMPLE PERCENT.
b6de691 [Michael Armbrust] Merge pull request #43 from liancheng/fixMakefile
1f6260d [Lian, Cheng] Fixed package name and test suite name in Makefile
5ae010f [Michael Armbrust] Merge pull request #42 from markhamstra/non-ascii
678341a [Mark Hamstra] Replaced non-ascii text
887f928 [Yin Huai] Merge remote-tracking branch 'upstream/master' into SerDeNew
1f7d00a [Reynold Xin] Merge pull request #41 from marmbrus/splitComponents
7588a57 [Michael Armbrust] Break into 3 major components and move everything into the org.apache.spark.sql package.
bc9a12c [Michael Armbrust] Move hive test files.
5720d2b [Lian, Cheng] Fixed comment typo
f0c3742 [Lian, Cheng] Refactored PhysicalOperation
f235914 [Lian, Cheng] Test case udf_regex and udf_like need BooleanWritable registered
cf691df [Lian, Cheng] Added the PhysicalOperation to generalize ColumnPrunings
2407a21 [Lian, Cheng] Added optimized logical plan to debugging output
a7ad058 [Michael Armbrust] Merge pull request #40 from marmbrus/includeGoldens
9329820 [Michael Armbrust] add golden answer files to repository
dce0593 [Michael Armbrust] move golden answer to the source code directory.
964368f [Michael Armbrust] Merge pull request #39 from marmbrus/lateralView
7785ee6 [Michael Armbrust] Tighten visibility based on comments.
341116c [Michael Armbrust] address comments.
0e6c1d7 [Reynold Xin] Merge pull request #38 from yhuai/parseDBNameInCTAS
2897deb [Michael Armbrust] fix scaladoc
7123225 [Yin Huai] Correctly parse the db name and table name in INSERT queries.
b376d15 [Michael Armbrust] fix newlines at EOF
5cc367c [Michael Armbrust] use berkeley instead of cloudbees
ff5ea3f [Michael Armbrust] new golden
db92adc [Michael Armbrust] more tests passing. clean up logging.
740febb [Michael Armbrust] Tests for tgfs.
0ce61b0 [Michael Armbrust] Docs for GenericHiveUdtf.
ba8897f [Michael Armbrust] Merge remote-tracking branch 'yin/parseDBNameInCTAS' into lateralView
dd00b7e [Michael Armbrust] initial implementation of generators.
ea76cf9 [Michael Armbrust] Add NoRelation to planner.
bea4b7f [Michael Armbrust] Add SumDistinct.
016b489 [Michael Armbrust] fix typo.
acb9566 [Michael Armbrust] Correctly type attributes of CTAS.
8841eb8 [Michael Armbrust] Rename Transform -> ScriptTransformation.
02ff8e4 [Yin Huai] Correctly parse the db name and table name in a CTAS query.
5e4d9b4 [Michael Armbrust] Merge pull request #35 from marmbrus/smallFixes
5479066 [Reynold Xin] Merge pull request #36 from marmbrus/partialAgg
8017afb [Michael Armbrust] fix copy paste error.
dc6353b [Michael Armbrust] turn off deprecation
cab1a84 [Michael Armbrust] Fix PartialAggregate inheritance.
883006d [Michael Armbrust] improve tests.
32b615b [Michael Armbrust] add override to asPartial.
e1999f9 [Yin Huai] Use Deserializer and Serializer instead of AbstractSerDe.
f94345c [Michael Armbrust] fix doc link
d8cb805 [Michael Armbrust] Implement partial aggregation.
ccdb07a [Michael Armbrust] Fix bug where averages of strings are turned into sums of strings.  Remove a blank line.
b4be6a5 [Michael Armbrust] better logging when applying rules.
67128b8 [Reynold Xin] Merge pull request #30 from marmbrus/complex
cb57459 [Michael Armbrust] blacklist machine specific test.
2f27604 [Michael Armbrust] Address comments / style errors.
389525d [Michael Armbrust] update golden, blacklist mr.
e3c10bd [Michael Armbrust] update whitelist.
44d343c [Michael Armbrust] Merge remote-tracking branch 'databricks/master' into complex
42ec4af [Michael Armbrust] improve complex type support in hive udfs/udafs.
ab5bff3 [Michael Armbrust] Support for get item of map types.
1679554 [Michael Armbrust] add toString for if and IS NOT NULL.
ab9a131 [Michael Armbrust] when UDFs fail they should return null.
25288d0 [Michael Armbrust] Implement [] for arrays and maps.
e7933e9 [Michael Armbrust] fix casting bug when working with fractional expressions.
010accb [Michael Armbrust] add tinyint to metastore type parser.
7a0f543 [Michael Armbrust] Avoid propagating types from unresolved nodes.
ac9d7de [Michael Armbrust] Resolve *s in Transform clauses.
692a477 [Michael Armbrust] Support for wrapping arrays to be written into hive tables.
92e4158 [Reynold Xin] Merge pull request #32 from marmbrus/tooManyProjects
9c06778 [Michael Armbrust] fix serialization issues, add JavaStringObjectInspector.
72a003d [Michael Armbrust] revert regex change
7661b6c [Michael Armbrust] blacklist machines specific tests
aa430e7 [Michael Armbrust] Update .travis.yml
e4def6b [Michael Armbrust] set dataType for HiveGenericUdfs.
5e54aa6 [Michael Armbrust] quotes for struct field names.
bbec500 [Michael Armbrust] update test coverage, new golden
3734a94 [Michael Armbrust] only quote string types.
3f9e519 [Michael Armbrust] use names w/ boolean args
5b3d2c8 [Michael Armbrust] implement distinct.
5b33216 [Michael Armbrust] work on decimal support.
2c6deb3 [Michael Armbrust] improve printing compatibility.
35a70fb [Michael Armbrust] multi-letter field names.
a9388fb [Michael Armbrust] printing for map types.
c3feda7 [Michael Armbrust] use toArray.
c654f19 [Michael Armbrust] Support for list and maps in hive table scan.
cf8d992 [Michael Armbrust] Use built in functions for creating temp directory.
1579eec [Michael Armbrust] Only cast unresolved inserts.
6420c7c [Michael Armbrust] Memoize the ordinal in the GetField expression.
da7ae9d [Michael Armbrust] Add boolean writable that was breaking udf_regexp test.  Not sure how this was passing before...
6709441 [Michael Armbrust] Evaluation for accessing nested fields.
dc6463a [Michael Armbrust] Support for resolving access to nested fields using "." notation.
d670e41 [Michael Armbrust] Print nested fields like hive does.
efa7217 [Michael Armbrust] Support for reading structs in HiveTableScan.
9c22b4e [Michael Armbrust] Support for parsing nested types.
82163e3 [Michael Armbrust] special case handling of partitionKeys when casting insert into tables
ea6f37f [Michael Armbrust] fix style.
7845364 [Michael Armbrust] deactivate concurrent test.
b649c20 [Michael Armbrust] fix test logging / caching.
1590568 [Michael Armbrust] add
19bfd74 [Michael Armbrust] store hive output in circular buffer
dfb67aa [Michael Armbrust] add test case
cb775ac [Michael Armbrust] get rid of SharkContext singleton
2de89d0 [Michael Armbrust] Merge pull request #13 from tnachen/master
63003e9 [Michael Armbrust] Fix spacing.
41b41f3 [Michael Armbrust] Only cast unresolved inserts.
6eb5960 [Michael Armbrust] Merge remote-tracking branch 'databricks/master' into udafs
5b7afd8 [Michael Armbrust] Merge pull request #10 from yhuai/exchangeOperator
b1151a8 [Timothy Chen] Fix load data regex
8e0931f [Michael Armbrust] Cast to avoid using deprecated hive API.
e079f2b [Timothy Chen] Add GenericUDAF wrapper and HiveUDAFFunction
45b334b [Yin Huai] fix comments
235cbb4 [Yin Huai] Merge remote-tracking branch 'upstream/master' into exchangeOperator
fc67b50 [Yin Huai] Check for a Sort operator with the global flag set instead of an Exchange operator with a RangePartitioning.
6015f93 [Michael Armbrust] Merge pull request #29 from rxin/style
271e483 [Michael Armbrust] Update build status icon.
d3a3d48 [Michael Armbrust] add testing to travis
807b2d7 [Michael Armbrust] check style and publish docs with travis
d20b565 [Michael Armbrust] fix if style
bce024d [Michael Armbrust] Merge remote-tracking branch 'databricks/master' into style Disable if brace checking as it errors in single line functional cases unlike the style guide.
d91e276 [Michael Armbrust] Remove dependence on HIVE_HOME for running tests.  This was done by moving all the hive query test (from branch-0.12) and data files into src/test/hive.  These are used by default when HIVE_HOME is not set.
f47c2f6 [Yin Huai] set outputPartitioning in BroadcastNestedLoopJoin
41bbee6 [Yin Huai] Merge remote-tracking branch 'upstream/master' into exchangeOperator
7e24436 [Reynold Xin] Removed dependency on JDK 7 (nio.file).
5c1e600 [Reynold Xin] Added hash code implementation for AttributeReference
7213a2c [Reynold Xin] style fix for Hive.scala.
08e4d05 [Reynold Xin] First round of style cleanup.
605255e [Reynold Xin] Added scalastyle checker.
61e729c [Lian, Cheng] Added ColumnPrunings strategy and test cases
2486fb7 [Lian, Cheng] Fixed spelling
8ee41be [Lian, Cheng] Minor refactoring
ebb56fa [Michael Armbrust] add travis config
4c89d6e [Reynold Xin] Merge pull request #27 from marmbrus/moreTests
d4f539a [Michael Armbrust] blacklist mr and user specific tests.
677eb07 [Michael Armbrust] Update test whitelist.
5dab0bc [Michael Armbrust] Merge pull request #26 from liancheng/serdeAndPartitionPruning
c263c84 [Michael Armbrust] Only push predicates into partitioned table scans.
ab77882 [Michael Armbrust] upgrade spark to RC5.
c98ede5 [Lian, Cheng] Response to comments from @marmbrus
83d4520 [Yin Huai] marmbrus's comments
70994a3 [Lian, Cheng] Revert unnecessary Scaladoc changes
9ebff47 [Yin Huai] remove unnecessary .toSeq
e811d1a [Yin Huai] markhamstra's comments
4802f69 [Yin Huai] The outputPartitioning of a UnaryNode inherits its child's outputPartitioning by default. Also, update the logic in AddExchange to avoid unnecessary shuffling operations.
040fbdf [Yin Huai] AddExchange is the only place to add Exchange operators.
9fb357a [Yin Huai] use getSpecifiedDistribution to create Distribution. ClusteredDistribution and OrderedDistribution do not take Nil as inptu expressions.
e9347fc [Michael Armbrust] Remove broken scaladoc links.
99c6707 [Michael Armbrust] upgrade spark
57799ad [Lian, Cheng] Added special treat for HiveVarchar in InsertIntoHiveTable
cb49af0 [Lian, Cheng] Fixed Scaladoc links
4e5e4d4 [Lian, Cheng] Added PreInsertionCasts to do necessary casting before insertion
111ffdc [Lian, Cheng] More comments and minor reformatting
9e0d840 [Lian, Cheng] Added partition pruning optimization
761bbb8 [Lian, Cheng] Generalized BindReferences to run against any query plan
04eb5da [Yin Huai] Merge remote-tracking branch 'upstream/master' into exchangeOperator
9dd3b26 [Michael Armbrust] Fix scaladoc.
6f44cac [Lian, Cheng] Made TableReader & HadoopTableReader private to catalyst
7c92a41 [Lian, Cheng] Added Hive SerDe support
ce5fdd6 [Yin Huai] Merge remote-tracking branch 'upstream/master' into exchangeOperator
2957f31 [Yin Huai] addressed comments on PR
907db68 [Michael Armbrust] Space after while.
04573a0 [Reynold Xin] Merge pull request #24 from marmbrus/binaryCasts
4e50679 [Reynold Xin] Merge pull request #25 from marmbrus/rowOrderingWhile
5bc1dc2 [Yin Huai] Merge remote-tracking branch 'upstream/master' into exchangeOperator
be1fff7 [Michael Armbrust] Replace foreach with while in RowOrdering. Fixes #23
fd084a4 [Michael Armbrust] implement casts binary <=> string.
0b31176 [Michael Armbrust] Merge pull request #22 from rxin/type
548e479 [Yin Huai] merge master into exchangeOperator and fix code style
5b11db0 [Reynold Xin] Added Void to Boolean type widening.
9e3d989 [Reynold Xin] Made HiveTypeCoercion.WidenTypes more clear.
9bb1979 [Reynold Xin] Merge pull request #19 from marmbrus/variadicUnion
a2beb38 [Michael Armbrust] Merge pull request #21 from liancheng/fixIssue20
b20a4d4 [Lian, Cheng] Fix issue #20
6d6cb58 [Michael Armbrust] add source links that point to github to the scala doc.
4285962 [Michael Armbrust] Remove temporary test cases
167162f [Michael Armbrust] more merge errors, cleanup.
e170ccf [Michael Armbrust] Improve documentation and remove some spurious changes that were introduced by the merge.
6377d0b [Michael Armbrust] Drop empty files, fix if ().
c0b0e60 [Michael Armbrust] cleanup broken doc links.
330a88b [Michael Armbrust] Fix bugs in AddExchange.
4f345f2 [Michael Armbrust] Remove SortKey, use RowOrdering.
043e296 [Michael Armbrust] Make physical union nodes variadic.
ece15e1 [Michael Armbrust] update unit tests
5c89d2e [Michael Armbrust] Merge remote-tracking branch 'databricks/master' into exchangeOperator Fix deprecated use of combineValuesByKey. Get rid of test where the answer is dependent on the plan execution width.
9804eb5 [Michael Armbrust] upgrade spark
053a371 [Michael Armbrust] Merge pull request #15 from marmbrus/orderedRow
5ab18be [Michael Armbrust] Merge remote-tracking branch 'databricks/master' into orderedRow
ca2ff68 [Michael Armbrust] Merge pull request #17 from marmbrus/unionTypes
bf9161c [Michael Armbrust] Merge pull request #18 from marmbrus/noSparkAgg
563053f [Michael Armbrust] Address @rxin's comments.
6537c66 [Michael Armbrust] Address @rxin's comments.
2a76fc6 [Michael Armbrust] add notes from @rxin.
685bfa1 [Michael Armbrust] fix spelling
69ed98f [Michael Armbrust] Output a single row for empty Aggregations with no grouping expressions.
7859a86 [Michael Armbrust] Remove SparkAggregate.  Its kinda broken and breaks RDD lineage.
fc22e01 [Michael Armbrust] whitelist newly passing union test.
3f547b8 [Michael Armbrust] Add support for widening types in unions.
53b95f8 [Michael Armbrust] coercion should not occur until children are resolved.
b892e32 [Michael Armbrust] Union is not resolved until the types match up.
95ab382 [Michael Armbrust] Use resolved instead of custom function.  This is better because some nodes override the notion of resolved.
81a109d [Michael Armbrust] fix link.
f143f61 [Michael Armbrust] Implement sampling.  Fixes a flaky test where the JVM notices that RAND as a Comparison method "violates its general contract!"
6cd442b [Michael Armbrust] Use numPartitions variable, fix grammar.
c800798 [Michael Armbrust] Add build status icon.
0cf5a75 [Michael Armbrust] Merge pull request #16 from marmbrus/filterPushDown
05d3a0d [Michael Armbrust] Refactor to avoid serializing ordering details with every row.
f2fdd77 [Michael Armbrust] fix required distribtion for aggregate.
658866e [Michael Armbrust] Pull back in changes made by @yhuai eliminating CoGroupedLocallyRDD.scala
583a337 [Michael Armbrust] break apart distribution and partitioning.
e8d41a9 [Michael Armbrust] Merge remote-tracking branch 'yin/exchangeOperator' into exchangeOperator
0ff8be7 [Michael Armbrust] Cleanup spurious changes and fix doc links.
73c70de [Yin Huai] add a first set of unit tests for data properties.
fbfa437 [Michael Armbrust] Merge remote-tracking branch 'databricks/master' into filterPushDown Minor doc improvements.
2b9d80f [Yin Huai] initial commit of adding exchange operators to physical plans.
fcbc03b [Michael Armbrust] Fix if ().
7b9080c [Michael Armbrust] Create OrderedRow class to allow ordering to be used by multiple operators.
b4adb0f [Michael Armbrust] Merge pull request #14 from marmbrus/castingAndTypes
b2a1ec5 [Michael Armbrust] add comment on how using numeric implicitly complicates spark serialization.
e286d20 [Michael Armbrust] address code review comments.
80d0681 [Michael Armbrust] fix scaladoc links.
de0c248 [Michael Armbrust] Print the executed plan in SharkQuery toString.
3413e61 [Michael Armbrust] Add mapChildren and withNewChildren methods to TreeNode.
404d552 [Michael Armbrust] Better exception when unbound attributes make it to evaluation.
fb84ae4 [Michael Armbrust] Refactor DataProperty into Distribution.
2abb0bc [Michael Armbrust] better debug messages, use exists.
098dfc4 [Michael Armbrust] Implement Long sorting again.
60f3a9a [Michael Armbrust] More aggregate functions out of the aggregate class to make things more readable.
a1ef62e [Michael Armbrust] Print the executed plan in SharkQuery toString.
dfce426 [Michael Armbrust] Add mapChildren and withNewChildren methods to TreeNode.
037a2ed [Michael Armbrust] Better exception when unbound attributes make it to evaluation.
ec90620 [Michael Armbrust] Support for Sets as arguments to TreeNode classes.
b21f803 [Michael Armbrust] Merge pull request #11 from marmbrus/goldenGen
83adb9d [Yin Huai] add DataProperty
5a26292 [Michael Armbrust] Rules to bring casting more inline with Hive semantics.
f0e0161 [Michael Armbrust] Move numeric types into DataTypes simplifying evaluator.  This can probably also be use for codegen...
6d2924d [Michael Armbrust] add support for If. Not integrated in HiveQL yet.
ccc4dbf [Michael Armbrust] Add optimization rule to simplify casts.
058ec15 [Michael Armbrust] handle more writeables.
ffa9f25 [Michael Armbrust] blacklist some more MR tests.
aa2239c [Michael Armbrust] filter test lines containing Owner:
f71a325 [Michael Armbrust] Update golden jar.
a3003ae [Michael Armbrust] Update makefile to use better sharding support.
568d150 [Michael Armbrust] Updates to white/blacklist.
8351f25 [Michael Armbrust] Add an ignored test to remind us we don't do empty aggregations right.
c4104ec [Michael Armbrust] Numerous improvements to testing infrastructure.  See comments for details.
09c6300 [Michael Armbrust] Add nullability information to StructFields.
5460b2d [Michael Armbrust] load srcpart by default.
3695141 [Michael Armbrust] Lots of parser improvements.
965ac9a [Michael Armbrust] Add expressions that allow access into complex types.
3ba53c9 [Michael Armbrust] Output type suffixes on AttributeReferences.
8777489 [Michael Armbrust] Initial support for operators that allow the user to specify partitioning.
e57f97a [Michael Armbrust] more decimal/null support.
e1440ed [Michael Armbrust] Initial support for function specific type conversions.
1814ed3 [Michael Armbrust] use childrenResolved function.
f2ec57e [Michael Armbrust] Begin supporting decimal.
6924e6e [Michael Armbrust] Handle NullTypes when resolving HiveUDFs
7fcfa8a [Michael Armbrust] Initial support for parsing unspecified partition parameters.
d0124f3 [Michael Armbrust] Correctly type null literals.
b65626e [Michael Armbrust] Initial support for parsing BigDecimal.
a90efda [Michael Armbrust] utility function for outputing string stacktraces.
7102f33 [Michael Armbrust] methods with side-effects should use ().
3ccaef7 [Michael Armbrust] add renaming TODO.
bc282c7 [Michael Armbrust] fix bug in getNodeNumbered
c8e89d5 [Michael Armbrust] memoize inputSet calculation.
6aefa46 [Michael Armbrust] Skip folding literals.
a72e540 [Michael Armbrust] Add IN operator.
04f885b [Michael Armbrust] literals are only non-nullable if they are not null.
35d2948 [Michael Armbrust] correctly order partition and normal attributes in hive relation output.
12fd52d [Michael Armbrust] support for sorting longs.
0606520 [Michael Armbrust] drop old comment.
859200a [Michael Armbrust] support for reading more types from the metastore.
1fedd18 [Michael Armbrust] coercion from null to numeric types
71e902d [Michael Armbrust] fix test cases.
cc06b6c [Michael Armbrust] Merge remote-tracking branch 'databricks/master' into interviewAnswer
8a8b521 [Reynold Xin] Merge pull request #8 from marmbrus/testImprovment
86355a6 [Michael Armbrust] throw error if there are unexpected join clauses.
c5842d2 [Michael Armbrust] don't throw an error when a select clause outputs multiple copies of the same attribute.
0e975ea [Michael Armbrust] parse bucket sampling as percentage sampling
a92919d [Michael Armbrust] add alter view as to native commands
f58d5a5 [Michael Armbrust] support for parsing SELECT DISTINCT
f0faa26 [Michael Armbrust] add sample and distinct operators.
ef7b943 [Michael Armbrust] add metastore support for float
e9f4588 [Michael Armbrust] fix > 100 char.
755b229 [Michael Armbrust] blacklist some ddl tests.
9ae740a [Michael Armbrust] blacklist more tests that require MR.
4cfc11a [Michael Armbrust] more test coverage.
0d9d56a [Michael Armbrust] add more native commands to parser
78d730d [Michael Armbrust] Load src test table on RESET.
8364ec2 [Michael Armbrust] whitelist all possible partition values.
b01468d [Michael Armbrust] support path rewrites when the query begins with a comment.
4c6b454 [Michael Armbrust] add option for recomputing the cached golden answer when tests fail.
4c5fb0f [Michael Armbrust] makefile target for building new whitelist.
4b6fed8 [Michael Armbrust] support for parsing both DESTINATION and INSERT_INTO.
516481c [Michael Armbrust] Ignore requests to explain native commands.
68aa2e6 [Michael Armbrust] Stronger type for Token extractor.
ca4ea26 [Michael Armbrust] Support for parsing UDF(*).
1aafea3 [Michael Armbrust] Configure partition whitelist in TestShark reset.
9627616 [Michael Armbrust] Use current database as default database.
9b02b44 [Michael Armbrust] Fix spelling error. Add failFast mode.
6f64cee [Michael Armbrust] don't line wrap string literal
eafaeed [Michael Armbrust] add type documentation
f54c94c [Michael Armbrust] make golden answers file a test dependency
5362365 [Michael Armbrust] push conditions into join
0d2388b [Michael Armbrust] Point at databricks hosted scaladoc.
73b29cd [Michael Armbrust] fix bad casting
9aa06c5 [Michael Armbrust] Merge pull request #7 from marmbrus/docFixes
7eff191 [Michael Armbrust] link all the expression names.
83227e4 [Michael Armbrust] fix scaladoc list syntax, add docs for some rules
9de6b74 [Michael Armbrust] fix language feature and deprecation warnings.
0b1960a [Michael Armbrust] Fix broken scala doc links / warnings.
b1acb36 [Michael Armbrust] Merge pull request #3 from yhuai/evalauteLiteralsInExpressions
01c00c2 [Michael Armbrust] new golden
5c14857 [Yin Huai] Merge remote-tracking branch 'upstream/master' into evalauteLiteralsInExpressions
b749b51 [Michael Armbrust] Merge pull request #5 from marmbrus/testCaching
66adceb [Michael Armbrust] Merge pull request #6 from marmbrus/joinWork
1a393da [Yin Huai] folded -> foldable
1e964ea [Yin Huai] update
a43d41c [Michael Armbrust] more tests passing!
8ca38d0 [Michael Armbrust] begin support for varchar / binary types.
ab8bbd1 [Michael Armbrust] parsing % operator
c16c8b5 [Michael Armbrust] case insensitive checking for hooks in tests.
3a90a5f [Michael Armbrust] simpler output when running a single test from the commandline.
5332fee [Yin Huai] Merge remote-tracking branch 'upstream/master' into evalauteLiteralsInExpressions
367fb9e [Yin Huai] update
0cd5cc6 [Michael Armbrust] add BIGINT cast parsing
61b266f [Michael Armbrust] comment for eliminate subqueries.
d72a5a2 [Michael Armbrust] add long to literal factory object.
b3bd15f [Michael Armbrust] blacklist more mr requiring tests.
e06fd38 [Michael Armbrust] black list map reduce tests.
8e7ce30 [Michael Armbrust] blacklist some env specific tests.
6250cbd [Michael Armbrust] Do not exit on test failure
b22b220 [Michael Armbrust] also look for cached hive test answers on the classpath.
b6e4899 [Yin Huai] formatting
e75c90d [Reynold Xin] Merge pull request #4 from marmbrus/hive12
5fabbec [Michael Armbrust] ignore partitioned scan test. scan seems to be working but there is some error about the table already existing?
9e190f5 [Michael Armbrust] drop unneeded ()
68b58c1 [Michael Armbrust] drop a few more tests.
b0aa400 [Michael Armbrust] update whitelist.
c99012c [Michael Armbrust] skip tests with hooks
db00ebf [Michael Armbrust] more types for hive udfs
dbc3678 [Michael Armbrust] update ghpages repo
138f53d [Yin Huai] addressed comments and added a space after a space after the defining keyword of every control structure.
6f954ee [Michael Armbrust] export the hadoop classpath when starting sbt, required to invoke hive during tests.
46bf41b [Michael Armbrust] add a makefile for priming the test answer cache in parallel.  usage: "make -j 8 -i"
8d47ed4 [Yin Huai] comment
2795f05 [Yin Huai] comment
e003728 [Yin Huai] move OptimizerSuite to the package of catalyst.optimizer
2941d3a [Yin Huai] Merge remote-tracking branch 'upstream/master' into evalauteLiteralsInExpressions
0bd1688 [Yin Huai] update
6a7bd75 [Michael Armbrust] fix partition column delimiter configuration.
e942da1 [Michael Armbrust] Begin upgrade to Hive 0.12.0.
b8cd7e3 [Michael Armbrust] Merge pull request #7 from rxin/moreclean
52864da [Reynold Xin] Added executeCollect method to SharkPlan.
f0e1cbf [Reynold Xin] Added resolved lazy val to LogicalPlan.
b367e36 [Reynold Xin] Replaced the use of ??? with UnsupportedOperationException.
38124bd [Yin Huai] formatting
2924468 [Yin Huai] add two tests for testing pre-order and post-order tree traversal, respectively
555d839 [Reynold Xin] More cleaning ...
d48d0e1 [Reynold Xin] Code review feedback.
aa2e694 [Yin Huai] Merge remote-tracking branch 'upstream/master' into evalauteLiteralsInExpressions
5c421ac [Reynold Xin] Imported SharkEnv, SharkContext, and HadoopTableReader to remove Shark dependency.
479e055 [Reynold Xin] A set of minor changes, including: - import order - limit some lines to 100 character wide - inline code comment - more scaladocs - minor spacing (i.e. add a space after if)
da16e45 [Reynold Xin] Merge pull request #3 from rxin/packagename
e36caf5 [Reynold Xin] Renamed to Rule.ruleName since name is used too frequently in the code base and is shadowed often by local scope.
72426ed [Reynold Xin] Rename shark2 package to execution.
0892153 [Reynold Xin] Merge pull request #2 from rxin/packagename
e58304a [Reynold Xin] Merge pull request #1 from rxin/gitignore
3f9fee1 [Michael Armbrust] rewrite push filter through join optimization.
c6527f5 [Reynold Xin] Moved the test src files into the catalyst directory.
c9777d8 [Reynold Xin] Put all source files in a catalyst directory.
019ea74 [Reynold Xin] Updated .gitignore to include IntelliJ files.
80ca4be [Timothy Chen] Address comments
0079392 [Michael Armbrust] support for multiple insert commands in a single query
75b5a01 [Michael Armbrust] remove space.
4283400 [Timothy Chen] Add limited predicate push down
e547e50 [Michael Armbrust] implement First.
e77c9b6 [Michael Armbrust] more work on unique join.
c795e06 [Michael Armbrust] improve star expansion
a26494e [Michael Armbrust] allow aliases to have qualifiers
d078333 [Michael Armbrust] remove extra space
a75c023 [Michael Armbrust] implement Coalesce
3a018b6 [Michael Armbrust] fix up docs.
ab6f67d [Michael Armbrust] import the string "null" as actual null.
5377c04 [Michael Armbrust] don't call dataType until checking if children are resolved.
191ce3e [Michael Armbrust] analyze rewrite test query.
60b1526 [Michael Armbrust] don't call dataType until checking if children are resolved.
2ab5a32 [Michael Armbrust] stop using uberjar as it has its own set of issues.
e42f75a [Michael Armbrust] Merge remote-tracking branch 'origin/master' into HEAD
c086a35 [Michael Armbrust] docs, spacing
c4060e4 [Michael Armbrust] cleanup
3b85462 [Michael Armbrust] more tests passing
bcfc8c5 [Michael Armbrust] start supporting partition attributes when inserting data.
c944a95 [Michael Armbrust] First aggregate expression.
1e28311 [Michael Armbrust] make tests execute in alpha order again
a287481 [Michael Armbrust] spelling
8492548 [Michael Armbrust] beginning of UNIQUEJOIN parsing.
a6ab6c7 [Michael Armbrust] add !=
4529594 [Michael Armbrust] draft of coalesce
70f253f [Michael Armbrust] more tests passing!
7349e7b [Michael Armbrust] initial support for test thrift table
d3c9305 [Michael Armbrust] fix > 100 char line
93b64b0 [Michael Armbrust] load test tables that are args to "DESCRIBE"
06b2aba [Michael Armbrust] don't be case sensitive when fixing load paths
6355d0e [Michael Armbrust] match actual return type of count with expected
cda43ab [Michael Armbrust] don't throw an exception when one of the join tables is empty.
fd4b096 [Michael Armbrust] fix casing of null strings as well.
4632695 [Michael Armbrust] support for megastore bigint
67b88cf [Michael Armbrust] more verbose debugging of evaluation return types
c680e0d [Michael Armbrust] Failed string => number conversion should return null.
2326be1 [Michael Armbrust] make getClauses case insensitive.
dac2786 [Michael Armbrust] correctly handle null values when going from string to numeric types.
045ac4b [Yin Huai] Merge remote-tracking branch 'upstream/master' into evalauteLiteralsInExpressions
fb5ddfd [Michael Armbrust] move ViewExamples to examples/
83833e8 [Michael Armbrust] more tests passing!
47c98d6 [Michael Armbrust] add query tests for like and hash.
1724c16 [Michael Armbrust] clear lines that contain last updated times.
cfd6bbc [Michael Armbrust] Quick skipping of tests that we can't even parse.
9b2642b [Michael Armbrust] make the blacklist support regexes
1d50af6 [Michael Armbrust] more datatypes, fix nonserializable instance variables in udfs
910e33e [Michael Armbrust] basic support for building an assembly jar.
d55bb52 [Michael Armbrust] add local warehouse/metastore to gitignore.
495d9dc [Michael Armbrust] Add an expression for when we decide to support LIKE natively instead of using the HIVE udf.
65f4e69 [Michael Armbrust] remove incorrect comments
0831a3c [Michael Armbrust] support for parsing some operator udfs.
6c27aa7 [Michael Armbrust] more cast parsing.
43db061 [Michael Armbrust] significant generalization of hive udf functionality.
3fe24ec [Michael Armbrust] better implementation of 3vl in Evaluate, fix some > 100 char lines.
e5690a6 [Michael Armbrust] add BinaryType
adab892 [Michael Armbrust] Clear out functions that are created during tests when reset is called.
d408021 [Michael Armbrust] support for printing out arrays in the output in the same form as hive (e.g., [e1, e1]).
8d5f504 [Michael Armbrust] Example of schema RDD using scala's dynamic trait, resulting in a more standard ORM style of usage.
21f0d91 [Michael Armbrust] Simple example of schemaRdd with scala filter function.
0daaa0e [Michael Armbrust] Promote booleans that appear in comparisons.
2b70abf [Michael Armbrust] true and false literals.
ef8b0a5 [Michael Armbrust] more tests.
14d070f [Michael Armbrust] add support for correctly extracting partition keys.
0afbe73 [Yin Huai] Merge remote-tracking branch 'upstream/master' into evalauteLiteralsInExpressions
69a0bd4 [Michael Armbrust] promote strings in predicates with number too.
3946e31 [Michael Armbrust] don't build strings unless assertion fails.
90c453d [Michael Armbrust] more tests passing!
6e6417a [Michael Armbrust] correct handling of nulls in boolean logic and sorting.
8000504 [Michael Armbrust] Improve type coercion.
9087152 [Michael Armbrust] fix toString of Not.
58b111c [Michael Armbrust] fix bad scaladoc tag.
d5c05c6 [Michael Armbrust] For now, ignore the big data benchmark tests when the data isn't there.
ac6376d [Michael Armbrust] Split out general shark query execution driver from test harness.
1d0ae1e [Michael Armbrust] Switch from IndexSeq[Any] to Row interface that will allow us unboxed access to primitive types.
d873b2b [Yin Huai] Remove numbers associated with test cases.
8545675 [Yin Huai] Merge remote-tracking branch 'upstream/master' into evalauteLiteralsInExpressions
b34a9eb [Michael Armbrust] Merge branch 'master' into filterPushDown
d1e7b8e [Michael Armbrust] Update
c8b1553 [Michael Armbrust] Update
9307ef9 [Michael Armbrust] update list of passing tests.
934c18c [Michael Armbrust] Filter out non-deterministic lines when comparing test answers.
a045c9c [Michael Armbrust] SparkAggregate doesn't actually support sum right now.
ae0024a [Yin Huai] update
cf80545 [Yin Huai] Merge remote-tracking branch 'origin/evalauteLiteralsInExpressions' into evalauteLiteralsInExpressions
21976ae [Yin Huai] update
b4999fe [Yin Huai] Merge remote-tracking branch 'upstream/filterPushDown' into evalauteLiteralsInExpressions
dedbf0c [Yin Huai] support Boolean literals
eaac9e2 [Yin Huai] explain the limitation of the current EvaluateLiterals
37817b5 [Yin Huai] add a comment to EvaluateLiterals.
468667f [Yin Huai] First draft of literal evaluation in the optimization phase. TreeNode has been extended to support transform in the post order. So, for an expression, we can evaluate literal from the leaf nodes of this expression tree. For an attribute reference in the expression node, we just leave it as is.
b1d1843 [Michael Armbrust] more work on big data benchmark tests.
cc9a957 [Michael Armbrust] support for creating test tables outside of TestShark
7d7fa9f [Michael Armbrust] support for create table as
5f54f03 [Michael Armbrust] parsing for ASC
d42b725 [Michael Armbrust] Sum of strings requires cast
34b30fa [Michael Armbrust] not all attributes need to be bound (e.g. output attributes that are contained in non-leaf operators.)
81659cb [Michael Armbrust] implement transform operator.
5cd76d6 [Michael Armbrust] break up the file based test case code for reuse
1031b65 [Michael Armbrust] support for case insensitive resolution.
320df04 [Michael Armbrust] add snapshot repo for databricks (has shark/spark snapshots)
b6f083e [Michael Armbrust] support for publishing scala doc to github from sbt
d9d18b4 [Michael Armbrust] debug logging implicit.
669089c [Yin Huai] support Boolean literals
ef3321e [Yin Huai] explain the limitation of the current EvaluateLiterals
73a05fd [Yin Huai] add a comment to EvaluateLiterals.
191eb7d [Yin Huai] First draft of literal evaluation in the optimization phase. TreeNode has been extended to support transform in the post order. So, for an expression, we can evaluate literal from the leaf nodes of this expression tree. For an attribute reference in the expression node, we just leave it as is.
80039cc [Yin Huai] Merge pull request #1 from yhuai/master
cbe1ca1 [Yin Huai] add explicit result type to the overloaded sideBySide
5c518e4 [Michael Armbrust] fix bug in test.
b50dd0e [Michael Armbrust] fix return type of overloaded method
05679b7 [Michael Armbrust] download assembly jar for easy compiling during interview.
8c60cc0 [Michael Armbrust] Update
03b9526 [Michael Armbrust] First draft of optimizer tests.
f392755 [Michael Armbrust] Add flatMap to TreeNode
6cbe8d1 [Michael Armbrust] fix bug in side by side, add support for working with unsplit strings
15a53fc [Michael Armbrust] more generic sum calculation and better binding of grouping expressions.
06749d0 [Michael Armbrust] add expression enumerations for query plan operators and recursive version of transform expression.
4b0a888 [Michael Armbrust] implement string comparison and more casts.
356b321 [Michael Armbrust] Update
3776395 [Michael Armbrust] Update
304d17d [Michael Armbrust] Create
b7d8be0 [Michael Armbrust] more tests passing.
b82481f [Michael Armbrust] add todo comment.
02e6dee [Michael Armbrust] add another test that breaks the harness to the blacklist.
cc5efe3 [Michael Armbrust] First draft of broadcast nested loop join with full outer support.
c43a259 [Michael Armbrust] comments
15ff448 [Michael Armbrust] better error message when a dsl test throws an exception
76ec650 [Michael Armbrust] fix join conditions
e10df99 [Michael Armbrust] Create new expr ids for local relations that exist more than once in a query plan.
91573a4 [Michael Armbrust] initial type promotion
e2ef4a5 [Michael Armbrust] logging
e43dc1e [Michael Armbrust] add string => int cast evaluation
f1f7e96 [Michael Armbrust] fix incorrect generation of join keys
2b27230 [Michael Armbrust] add depth based subtree access
0f6279f [Michael Armbrust] broken tests.
389bc0b [Michael Armbrust] support for partitioned columns in output.
12584f4 [Michael Armbrust] better errors for missing clauses. support for matching multiple clauses with the same name.
b67a225 [Michael Armbrust] better errors when types don't match up.
9e74808 [Michael Armbrust] add children resolved.
6d03ce9 [Michael Armbrust] defaults for unresolved relation
2469b00 [Michael Armbrust] skip nodes with unresolved children when doing coersions
be5ae2c [Michael Armbrust] better resolution logging
cb7b5af [Michael Armbrust] views example
420e05b [Michael Armbrust] more tests passing!
6916c63 [Michael Armbrust] Reading from partitioned hive tables.
a1245f9 [Michael Armbrust] more tests passing
956e760 [Michael Armbrust] extended explain
5f14c35 [Michael Armbrust] more test tables supported
175c43e [Michael Armbrust] better errors for parse exceptions
480ade5 [Michael Armbrust] don't use partial cached results.
8a9d21c [Michael Armbrust] fix evaluation
7aee69c [Michael Armbrust] parsing for joins, boolean logic
7fcf480 [Michael Armbrust] test for and logic
3ea9b00 [Michael Armbrust] don't use simpleString if there are no new lines.
6902490 [Michael Armbrust] fix boolean logic evaluation
4d5eba7 [Michael Armbrust] add more dsl for expression arithmetic and boolean logic
8b2a2ee [Michael Armbrust] more tests passing!
ad1f3b4 [Michael Armbrust] toString for null literals
a5c0a1b [Michael Armbrust] more test harness improvements: * regex whitelist * side by side answer comparison (still needs formatting work)
60ec19d [Michael Armbrust] initial support for udfs
c45b440 [Michael Armbrust] support for is (not) null and boolean logic
7f4a1dc [Michael Armbrust] add NoRelation logical operator
72e183b [Michael Armbrust] support for null values in tree node args.
ad596d2 [Michael Armbrust] add sc to Union's otherCopyArgs
e5c9d1a [Michael Armbrust] use nonEmpty
dcc4fe1 [Michael Armbrust] support for src1 test table.
c78b587 [Michael Armbrust] casting.
75c3f3f [Michael Armbrust] add support for logging with scalalogging.
da2c011 [Michael Armbrust] make it more obvious when results are being truncated.
96b73ba [Michael Armbrust] more docs in TestShark
18524fd [Michael Armbrust] add method to SharkSqlQuery for directly executing the same query on hive.
e6d063b [Michael Armbrust] more join tests.
664c1c3 [Michael Armbrust] make parsing of function names case insensitive.
0967d4e [Michael Armbrust] fix hardcoded path to hiveDevHome.
1a6db68 [Michael Armbrust] spelling
7638cb4 [Michael Armbrust] simple join execution with dsl tests.  no hive tests yes.
859d4c9 [Michael Armbrust] better argString printing of nested trees.
fc53615 [Michael Armbrust] add same instance comparisons for tree nodes.
a026e6b [Michael Armbrust] move out hive specific operators
fff4d1c [Michael Armbrust] add simple query execution debugging
e2120ab [Michael Armbrust] sorting for strings
da06eb6 [Michael Armbrust] Parsing for sortby and joins
9eb5c5e [Michael Armbrust] override equality in Attribute references to compare exprId.
8eb2460 [Michael Armbrust] add system property to override whitelist.
88124bb [Michael Armbrust] make strategy evaluation lazy.
74a3a21 [Michael Armbrust] implement outputSet
d25b171 [Michael Armbrust] Add AND and OR expressions
67f0a4a [Michael Armbrust] dsl improvements: string to attribute, subquery, unionAll
12acf0a [Michael Armbrust] add .DS_Store for macs
f7da6ce [Michael Armbrust] add agg with grouping expr in select test
36805b3 [Michael Armbrust] pull out and improve aggregation
75613e1 [Michael Armbrust] better evaluations failure messages.
4789a35 [Michael Armbrust] weaken type since its hard to create pure references.
e89dd36 [Michael Armbrust] no newline for online trees
d0590d4 [Michael Armbrust] include stack trace for catalyst failures.
081c0d9 [Michael Armbrust] more generic computation of agg functions.
31af3a0 [Michael Armbrust] fail when clauses are unhandeled in the parser
ecd45b2 [Michael Armbrust] Add more passing tests.
97d5419 [Michael Armbrust] fix alignment.
565cc13 [Michael Armbrust] make the canary query optional.
a95e65c [Michael Armbrust] support for resolving qualified attribute references.
e1dfa0c [Michael Armbrust] better error reporting for comparison tests when hive works but catalyst fails.
4640a0b [Michael Armbrust] handle test tables when database is specified.
bef12e3 [Michael Armbrust] Add Subquery node and trivial optimizer to remove it after analysis.
fec5158 [Michael Armbrust] add hive / idea files to .gitignore
3f97ffe [Michael Armbrust] Rename Hive => HiveQl
656b836 [Michael Armbrust] Support for parsing select clause aliases.
3ca7414 [Michael Armbrust] StopAfter needs otherCopyArgs.
3ffde66 [Michael Armbrust] When the child of an alias is unresolved it should return an unresolved attribute instead of throwing an exception.
8cbef8a [Michael Armbrust] spelling
aa8c37c [Michael Armbrust] Better toString for SortOrder
1bb8b45 [Michael Armbrust] fix error message for UnresolvedExceptions
a2e0327 [Michael Armbrust] add a bunch of tests.
4a3e1ea [Michael Armbrust] docs and use shark for data loading.
339bb8f [Michael Armbrust] better docs, Not support
1d7b2d9 [Michael Armbrust] Add NaN conversions.
46a2534 [Michael Armbrust] only run canary query on failure.
8996066 [Michael Armbrust] remove protected from makeCopy
53bcf41 [Michael Armbrust] testing improvements: * reset hive vars * delete indexes and tables * delete database * reset to use default database * record tests that pass
04a372a [Michael Armbrust] add a flag for running all tests.
3b2235b [Michael Armbrust] More general implementation of arithmetic.
edd7795 [Michael Armbrust] More testing improvements: * Check that results match for native commands * Ensure explain commands can be planned * Cache hive "golden" results
da6c577 [Michael Armbrust] add string <==> file utility functions.
3adf5ca [Michael Armbrust] Initial support for groupBy and count.
7bcd8a4 [Michael Armbrust] Improvements to comparison tests: * Sort answer when query doesn't contain an order by. * Display null values the same as Hive. * Print full query results in easy to read format when they differ.
a52e7c9 [Michael Armbrust] Transform children that are present in sequences of the product.
d66ba7e [Michael Armbrust] drop printlns.
88f2efd [Michael Armbrust] Add sum / count distinct expressions.
05adedc [Michael Armbrust] rewrite relative paths when loading data in TestShark
07784b3 [Michael Armbrust] add support for rewriting paths and running 'set' commands.
b8a9910 [Michael Armbrust] quote tests passing.
8e5e267 [Michael Armbrust] handle aliased select expressions.
4286a96 [Michael Armbrust] drop debugging println
ac34aeb [Michael Armbrust] proof of concept for hive ast transformations.
2238b00 [Michael Armbrust] better error when makeCopy functions fails due to incorrect arguments
ff1eab8 [Michael Armbrust] start trying to make insert into hive table more general.
74a6337 [Michael Armbrust] use fastEquals when doing transformations.
1184a23 [Michael Armbrust] add native test for escapes.
b972b18 [Michael Armbrust] create BaseRelation class
fa6bce9 [Michael Armbrust] implement union
6391a87 [Michael Armbrust] count aggregate.
d47c317 [Michael Armbrust] add unary minus, more tests passing.
c7114e4 [Michael Armbrust] first draft of star expansion.
044c43d [Michael Armbrust] better support for numeric literal parsing.
1d0f072 [Michael Armbrust] use native drop table as it doesn't appear to fail when the "table" is actually a view.
61503c5 [Michael Armbrust] add cached toRdd
2036883 [Michael Armbrust] skip explain queries when testing.
ebac4b1 [Michael Armbrust] fix bug in sort reference calculation
ca0dee0 [Michael Armbrust] docs.
1ee0471 [Michael Armbrust] string literal parsing.
357278b [Michael Armbrust] add limit support
9b3e479 [Michael Armbrust] creation of string literals.
02efa30 [Michael Armbrust] alias evaluation
cb68b33 [Michael Armbrust] parsing for random sample in hive ql.
126dd36 [Michael Armbrust] include query plans in failure output
bb59ae9 [Michael Armbrust] doc fixes
7e68286 [Michael Armbrust] fix confusing naming
768bb25 [Michael Armbrust] handle errors in shark query toString
829c3ce [Michael Armbrust] Auto loading of test data on demand. Add reset method to test shark.  Make test shark a singleton to avoid weirdness with the hive megastore.
ad02e41 [Michael Armbrust] comment jdo dependency
7bc89fe [Michael Armbrust] add collect to TreeNode.
438cf74 [Michael Armbrust] create explicit treeString function in addition to toString override. docs.
09679ee [Michael Armbrust] fix bug in TreeNode foreach
2930b27 [Michael Armbrust] more specific name for del query tests.
8842549 [Michael Armbrust] docs.
da81f81 [Michael Armbrust] Implementation and tests for simple AVG query in Hive SQL.
a8969b9 [Michael Armbrust] Factor out hive query comparison test framework.
1a7efb0 [Michael Armbrust] specialize spark aggregate for global aggregations.
a36dd9a [Michael Armbrust] evaluation for other > data types.
cae729b [Michael Armbrust] remove unnecessary lazy vals.
d8e12af [Michael Armbrust] docs
3a60d67 [Michael Armbrust] implement average, placeholder for count
f05c106 [Michael Armbrust] checkAnswer handles single row results.
2730534 [Michael Armbrust] implement inputSet
a9aa79d [Michael Armbrust] debugging for sort exec
8bec3c9 [Michael Armbrust] better tree makeCopy when there are two constructors.
554b4b2 [Michael Armbrust] BoundAttribute pretty printing.
754f5fa [Michael Armbrust] dsl for setting nullability
a206d7a [Michael Armbrust] clean up query tests.
84ad6ef [Michael Armbrust] better sort implementation and tests.
de24923 [Michael Armbrust] add double type.
9611a2c [Michael Armbrust] literal creation for doubles.
7358313 [Michael Armbrust] sort order returns child type.
b544715 [Michael Armbrust] implement eval for rand, and > for doubles
7013bad [Michael Armbrust] asc, desc should work for expressions and unresolved attributes (symbols)
1c1a35e [Michael Armbrust] add simple Rand expression.
3ca51de [Michael Armbrust] add orderBy to dsl
7ae41ab [Michael Armbrust] more literal implicit conversions
b18b675 [Michael Armbrust] First cut at native query tests for shark.
d392e29 [Michael Armbrust] add toRdd implicit conversion for logical plans in TestShark.
5eac895 [Michael Armbrust] better error when descending is specified.
2b16f86 [Michael Armbrust] add todo
e527bb8 [Michael Armbrust] remove arguments to binary predicate constructor as they seem to break serialization
9dde3c8 [Michael Armbrust] add project and filter operations.
ad9037b [Michael Armbrust] Add support for local relations.
6227143 [Michael Armbrust] evaluation of Equals.
7526290 [Michael Armbrust] BoundReference should also be an Attribute.
bd33e26 [Michael Armbrust] more documentation
5de0ea3 [Michael Armbrust] Move all shark specific into a separate package.  Lots of documentation improvements.
0ae292b [Michael Armbrust] implement calculation of sort expressions.
9fd5011 [Michael Armbrust] First cut at expression evaluation.
6259e3a [Michael Armbrust] cleanup
787e5a2 [Michael Armbrust] use fastEquals
f90da36 [Michael Armbrust] better printing of optimization exceptions
b05dd67 [Michael Armbrust] Application of rules to fixed point.
bb2e0db [Michael Armbrust] pretty print for literals.
1ec3287 [Michael Armbrust] Add extractor for IntegerLiterals.
d3a3687 [Michael Armbrust] add fastEquals
2b4935b [Michael Armbrust] set sbt.version explicitly
46dfd7f [Michael Armbrust] first cut at checking answer for HiveCompatability tests.
c79f2fd [Michael Armbrust] insert operator should return an empty rdd.
14c22ec [Michael Armbrust] implement sorting when the sort expression is the first attribute of the input.
ae7b4c3 [Michael Armbrust] remove implicit dependencies.  now compiles without copying things into lib/ manually.
84082f9 [Michael Armbrust] add sbt binaries and scripts
15371a8 [Michael Armbrust] First draft of simple Hive DDL parser.
063bf44 [Michael Armbrust] Periods should end all comments.
e1f7f4c [Michael Armbrust] Remove "NativePlaceholder" hack.
ed3633e [Michael Armbrust] start consolidating Hive/Shark specific code. first hive compatibility test case passing!
b34a770 [Michael Armbrust] Add data sink strategy, make strategy application a little more robust.
e7174ec [Michael Armbrust] fix schema, add docs, make helper method protected.
26f410a [Michael Armbrust] physical traits should extend PhysicalPlan.
dc72469 [Michael Armbrust] beginning of hive compatibility testing framework.
0763490 [Michael Armbrust] support for hive native command pass-through.
d8a924f [Michael Armbrust] scaladoc
29a7163 [Michael Armbrust] Insert into hive table physical operator.
633cebc [Michael Armbrust] better error message when there is no appropriate planning strategy.
59ac444 [Michael Armbrust] add unary expression
3aa1b28 [Michael Armbrust] support for table names in the form 'database.tableName'
665f7d0 [Michael Armbrust] add logical nodes for hive data sinks.
64d2923 [Michael Armbrust] Add classes for representing sorts.
f72b7ce [Michael Armbrust] first trivial end to end query execution.
5c7d244 [Michael Armbrust] first draft of references implementation.
7bff274 [Michael Armbrust] point at new shark.
c7cd57f [Michael Armbrust] docs for util function.
910811c [Michael Armbrust] check each item of the sequence
ef21a0b [Michael Armbrust] line up comments.
4b765d5 [Michael Armbrust] docs, drop println
6f9bafd [Michael Armbrust] empty output for unresolved relation to avoid exception in resolution.
a703c49 [Michael Armbrust] this order works better until fixed point is implemented.
ec1d7c0 [Michael Armbrust] Simple attribute resolution.
069df02 [Michael Armbrust] parsing binary predicates
a1cf754 [Michael Armbrust] add joins and equality.
3f5bc98 [Michael Armbrust] add optiq to sbt.
54f3460 [Michael Armbrust] initial optiq parsing.
d9161ce [Michael Armbrust] add join operator
1e423eb [Michael Armbrust] placeholders in LogicalPlan, docs
24ef6fb [Michael Armbrust] toString for alias.
ae7d776 [Michael Armbrust] add nullability changing function
d49dc02 [Michael Armbrust] scaladoc for named exprs
7c45dd7 [Michael Armbrust] pretty printing of trees.
78e34bf [Michael Armbrust] simple git ignore.
7ba19be [Michael Armbrust] First draft of interface to hive metastore.
7e7acf0 [Michael Armbrust] physical placeholder.
1c11136 [Michael Armbrust] first draft of error handling / plans for debugging.
3766a41 [Michael Armbrust] rearrange utility functions.
7fb3d5e [Michael Armbrust] docs and equality improvements.
45da47b [Michael Armbrust] flesh out plans and expressions a little. first cut at named expressions.
002d4d4 [Michael Armbrust] default to no alias.
be25003 [Michael Armbrust] add repl initialization to sbt.
0608a00 [Michael Armbrust] tighten public interface
a1a8b38 [Michael Armbrust] test that ids don't change for no-op transforms.
daa71ca [Michael Armbrust] foreach, maps, and scaladoc
6a158cb [Michael Armbrust] simple transform working.
db0299f [Michael Armbrust] basic analysis of relations minus transform function.
f74c4ee [Michael Armbrust] parsing a simple query.
08e4f57 [Michael Armbrust] upgrade scala include shark.
d3c6404 [Michael Armbrust] initial commit
Cheng Lian liancheng referenced this pull request from a commit in liancheng/scala-dist
Patrick Wendell pwendell SPARK-1094 Support MiMa for reporting binary compatibility accross ve…

This adds some changes on top of the initial work by @scrapcodes in #20:

The goal here is to do automated checking of Spark commits to determine whether they break binary compatibility.

1. Special case for inner classes of package-private objects.
2. Made tools classes accessible when running `spark-class`.
3. Made some declared types in MLLib more general.
4. Various other improvements to exclude-generation script.
5. In-code documentation.

Author: Patrick Wendell <>
Author: Prashant Sharma <>
Author: Prashant Sharma <>

Closes #207 from pwendell/mima and squashes the following commits:

22ae267 [Patrick Wendell] New binary changes after upmerge
6c2030d [Patrick Wendell] Merge remote-tracking branch 'apache/master' into mima
3666cf1 [Patrick Wendell] Minor style change
0e0f570 [Patrick Wendell] Small fix and removing directory listings
647c547 [Patrick Wendell] Reveiw feedback.
c39f3b5 [Patrick Wendell] Some enhancements to binary checking.
4c771e0 [Prashant Sharma] Added a tool to generate mima excludes and also adapted build to pick automatically.
b551519 [Prashant Sharma] adding a new exclude after rebasing with master
651844c [Prashant Sharma] Support MiMa for reporting binary compatibility accross versions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on May 14, 2012
  1. soc

    Updates Scala specification for 2.10

    soc authored
    - Removed Octal literals
    - Disallowed FP literals without digit after dot
    - Removed val in for comprehension value definitions (Fixes SI-4918)
    - Added ## to the list of non-NPE-throwing Null methods (Fixes SI-4311)
    - Adapted some wording around AnyRef/AnyVal
    - Removed primitive type aliases (Fixes SI-4628)
    - Casting of null (Fixes second part of SI-4437)
    - Removed ScalaObject from the spec (Closes SI-4648)
    - Merged the changes of the PDF-only Scala-2.9-Draft manually
This page is out of date. Refresh to see the latest.
50 documentation/src/reference/ImplementationStatus.tex
@@ -1,50 +0,0 @@
-The present Scala compiler does not yet implement all of the Scala
-specification. Its currently existing omissions and deviations are
-listed below. We are working on a refined implementation that
-addresses these issues.
-Unicode support is still limited. At present we only permit Unicode
-encodings \verb@\uXXXX@ in strings and backquote-enclosed identifiers.
-To define or access a Unicode identifier, you need to put it in
-backquotes and use the \verb@\uXXXX@ encoding.
-The unicode operator ``$\Rightarrow$''
-(\sref{sec:idents}) is not yet recognized; you need to use the two
-character ASCII equivalent ``\code{=>}'' instead.
-The current implementation does not yet support run-time types.
-All types are erased (\sref{sec:erasure}) during compilation. This means that
-the following operations give potentially wrong results.
-Type tests and type casts to parameterized types. Here it is only tested
-that a value is an instance of the given top-level type constructor.
-Type tests and type casts to type parameters and abstract types. Here
-it is only tested that a value is an instance of the type parameter's upper bound.
-Polymorphic array creation. If \code{t} is a type variable or abstract type, then
-\code{new Array[t]} will yield an array of the upper bound of \code{t}.
-Return expressions are not yet permitted inside an anonymous function
-or inside a call-by-name argument (i.e.\ a function argument corresponding to a
-\code{def} parameter).
-Members of the empty package (\sref{sec:packagings}) cannot yet be
-accessed from other source files. Hence, all library classes and
-objects have to be in some package.
-At present, auxiliary constructors (\sref{sec:constr-defs}) are only permitted
-for monomorphic classes.
-The \code{Array} class supports as yet only a restricted set of
-operations as given in \sref{cls:array}. It is planned to extend that
-interface. In particular, arrays will implement the \code{scala.Seq}
-trait as well as the methods needed to support for-comprehensions.
-At present, all classes used as mixins must be accessible to the Scala
-compiler in source form.
163 documentation/src/reference/RationalePart.tex
@@ -1,163 +0,0 @@
-There are hundreds of programming languages in active use, and many
-more are being designed each year. It is therefore hard to justify the
-development of yet another language. Nevertheless, this is what we
-attempt to do here. Our argument is based on two claims:
-\item[] {\em Claim 1:} The raise in importance of web services and
-other distributed software is a fundamental paradigm
-shift in programming. It is comparable in scale to the shift 20 years ago
-from character-oriented to graphical user interfaces.
-\item[] {\em Claim 2:} That paradigm shift will provide demand
-for new programming languages, just as graphical user interfaces
-promoted the adoption of object-oriented languages.
-For the last 20 years, the most common programming model was
-object-oriented: System components are objects, and computation is
-done by method calls. Methods themselves take object references as
-parameters. Remote method calls let one extend this programming model
-to distributed systems. The problem of this model is that it does not
-scale up very well to wide-scale networks where messages can be
-delayed and components may fail. Web services address the message
-delay problem by increasing granularity, using method calls with
-larger, structured arguments, such as XML trees. They address the
-failure problem by using transparent replication and avoiding server
-state. Conceptually, they are {\em tree transformers} that consume
-incoming message documents and produce outgoing ones. \comment{ To
-back up the first claim, one observes that web services and other
-distributed software increasingly tend to communicate using structured
-or semi-structured data. A typical example is the use of XML to
-describe data managed by applications as well as the messages between
-applications. This tends to affect the role of a program in a
-fundamental way. Previously, programs could be seen as objects that
-reacted to method calls and in turn called methods of other
-objects. Some of these method calls might originate from users while
-others might originate from other computers via remote invocations.
-These method calls have simple unstructured parameters or object
-references as arguments. Web services, on the other hand, communicate
-with each other by transmitting asynchronous messages that carry
-structured documents, usually in XML format. Programs then
-conceptually become {\em tree transformers} that consume incoming
-message documents and produce outgoing ones. }
-Why should this have an effect on programming languages? There are at
-least two reasons: First, today's object-oriented languages are not
-very good at analyzing and transforming XML trees. Because such trees
-usually contain data but no methods, they have to be decomposed and
-constructed from the ``outside'', that is from code which is external
-to the tree definition itself. In an object-oriented language, the
-ways of doing so are limited. The most common solution \cite{w3c:dom} is
-to represent trees in a generic way, where all tree nodes are values
-of a common type. This makes it easy to write generic traversal
-functions, but forces applications to operate on a very low conceptual
-level, which often loses important semantic distinctions present in
-the XML data. More semantic precision is obtained if different
-internal types model different kinds of nodes. But then tree
-decompositions require the use of run-time type tests and type casts
-to adapt the treatment to the kind of node encountered. Such type
-tests and type casts are generally not considered good object-oriented
-style. They are rarely efficient, nor easy to use.
-By contrast, tree transformation is the natural domain of functional
-languages. Their algebraic data types, pattern matching and
-higher-order functions make these languages ideal for the task. It's
-no wonder, then, that specialized languages for transforming XML data
-such as XSLT are functional.
-Another reason why functional language constructs are attractive for
-web-services is that mutable state is problematic in this setting.
-Components with mutable state are harder to replicate or to restore
-after a failure. Data with mutable state is harder to cache than
-immutable data. Functional language constructs make it relatively easy
-to construct components without mutable state.
-Many web services are constructed by combining different languages.
-For instance, a service might use XSLT to handle document
-transformation, XQuery for database access, and Java for the
-``business logic''. The downside of this approach is that the
-necessary amount of cross-language glue can make applications
-cumbersome to write, verify, and maintain. A particular problem is
-that cross-language interfaces are usually not statically typed.
-Hence, the benefits of a static type system are missing where they are
-needed most -- at the join points of components written in different
-Conceivably, the glue problem could be addressed by a ``multi-paradigm''
-language that would express object-oriented, concurrent, as well
-as functional aspects of an application. But one needs to be careful
-not to simply replace cross-language glue by awkward interfaces
-between different paradigms within the language itself. Ideally, one
-would hope for a fusion which unifies concepts found in different
-paradigms instead of an agglutination, which merely includes them side
-by side. This fusion is what we try to achieve with Scala
-\footnote{Scala stands for ``Scalable Language''. The term means
- ``Stairway'' in Italian}.
-Scala is both an object-oriented and functional language. It is a
-pure object-oriented language in the sense that every value is an
-object. Types and behavior of objects are described by
-classes. Classes can be composed using mixin composition. Scala is
-designed to work seamlessly with mainstream object-oriented languages,
-in particular Java and C\#.
-Scala is also a functional language in the sense that every function
-is a value. Nesting of function definitions and higher-order functions
-are naturally supported. Scala also supports a general notion of
-pattern matching which can model the algebraic types used in many
-functional languages. Furthermore, this notion of pattern matching
-naturally extends to the processing of XML data.
-The design of Scala is driven by the desire to unify object-oriented
-and functional elements. Here are three examples how this is achieved:
-Since every function is a value and every value is an object, it
-follows that every function in Scala is an object. Indeed, there is a
-root class for functions which is specialized in the Scala standard
-library to data structures such as arrays and hash tables.
-Data structures in many functional languages are defined using
-algebraic data types. They are decomposed using pattern matching.
-Object-oriented languages, on the other hand, describe data with class
-hierarchies. Algebraic data types are usually closed, in that the
-range of alternatives of a type is fixed when the type is defined. By
-contrast, class hierarchies can be extended by adding new leaf
-classes. Scala adopts the object-oriented class hierarchy scheme for
-data definitions, but allows pattern matching against values coming
-from a whole class hierarchy, not just values of a single type.
-This can express both closed and extensible data types, and also
-provides a convenient way to exploit run-time type information in
-cases where static typing is too restrictive.
-Module systems of functional languages such as SML or Caml excel in
-abstraction; they allow very precise control over visibility of names
-and types, including the ability to partially abstract over types. By
-contrast, object-oriented languages excel in composition; they offer
-several composition mechanisms lacking in module systems, including
-inheritance and unlimited recursion between objects and classes.
-Scala unifies the notions of object and module, of module signature
-and interface, as well as of functor and class. This combines the
-abstraction facilities of functional module systems with the
-composition constructs of object-oriented languages. The unification
-is made possible by means of a new type system based on path-dependent
-types \cite{odersky-et-al:fool10}.
-There are several other languages that try to bridge the gap between
-the functional and object oriented
-paradigms. Smalltalk\cite{goldberg-robson:smalltalk-language},
-Python\cite{rossum:python}, or Ruby\cite{matsumtoto:ruby} come to
-mind. Unlike these languages, Scala has an advanced static type
-system, which contains several innovative constructs. This aspect
-makes the Scala definition a bit more complicated than those of the
-languages above. On the other hand, Scala enjoys the robustness,
-safety and scalability benefits of strong static typing. Furthermore,
-Scala incorporates recent advances in type inference, so that
-excessive type annotations in user programs can usually be avoided.
-% rest of this report is structured as follows. Chapters
-%\ref{sec:simple-examples} to \ref{sec:concurrency} give an informal overview of
-%Scala by means of a sequence of program examples. The remaining
-%chapters contain the language definition. The definition is formulated
-%in prose but tries to be precise.
129 documentation/src/reference/ReferencePart.tex
@@ -409,13 +409,11 @@ \section{Literals}\label{sec:literals}
\subsection{Integer Literals}
-integerLiteral ::= (decimalNumeral | hexNumeral | octalNumeral) [`L' | `l']
+integerLiteral ::= (decimalNumeral | hexNumeral) [`L' | `l']
decimalNumeral ::= `0' | nonZeroDigit {digit}
hexNumeral ::= `0' `x' hexDigit {hexDigit}
-octalNumeral ::= `0' octalDigit {octalDigit}
digit ::= `0' | nonZeroDigit
nonZeroDigit ::= `1' | $\cdots$ | `9'
-octalDigit ::= `0' | $\cdots$ | `7'
Integer literals are usually of type \lstinline@Int@, or of type
\lstinline@Long@ when followed by a \lstinline@L@ or
@@ -441,13 +439,13 @@ \subsection{Integer Literals}
Here are some integer literals:
-0 21 0xFFFFFFFF 0777L
+0 21 0xFFFFFFFF -42L
\subsection{Floating Point Literals}
-floatingPointLiteral ::= digit {digit} `.' {digit} [exponentPart] [floatType]
+floatingPointLiteral ::= digit {digit} `.' digit {digit} [exponentPart] [floatType]
| `.' digit {digit} [exponentPart] [floatType]
| digit {digit} exponentPart [floatType]
| digit {digit} [exponentPart] floatType
@@ -472,16 +470,10 @@ \subsection{Floating Point Literals}
-The phrase `\lstinline@1.toString@' parses as three different tokens:
-`\lstinline@1@', `\lstinline@.@', and `\lstinline@toString@'. On the
-other hand, if a space is inserted after the period, the phrase
-`\lstinline@1. toString@' parses as the floating point literal
-`\lstinline@1.@' followed by the identifier `\lstinline@toString@'.
+`\lstinline@1.@' is not a valid floating point literal, because the
+mandatory digit after the `\lstinline@.@' is missing. For instance
+the phrase `\lstinline@1.toString@' parses as three different tokens:
+`\lstinline@1@', `\lstinline@.@', and `\lstinline@toString@'.
\subsection{Boolean Literals}
@@ -544,11 +536,11 @@ \subsubsection*{Multi-Line String Literals}
A multi-line string literal is a sequence of characters enclosed in
triple quotes ~\lstinline@""" ... """@. The sequence of characters is
-arbitrary, except that it may contain three or more consuctive quote characters
-only at the very end. Characters
-must not necessarily be printable; newlines or other
-control characters are also permitted. Unicode escapes work as everywhere else, but none
-of the escape sequences in (\sref{sec:escapes}) is interpreted.
+arbitrary, except that it may contain three or more consecutive quote
+characters only at the very end. Characters must not necessarily be
+printable; newlines or other control characters are also permitted.
+Unicode escapes work as everywhere else, but none of the escape sequences
+in (\sref{sec:escapes}) is interpreted.
\example Here is a multi-line string literal:
@@ -1007,7 +999,7 @@ \subsection{Tuple Types}\label{sec:tuple-types}
extends Product$n$[T1, ..., T$n$] {}
trait Product$n$[+T1, +T2, +T$n$] {
- override def arity = $n$
+ override def productArity = $n$
def _1: T1
def _$n$:T$n$
@@ -1929,7 +1921,7 @@ \section{Value Declarations and Definitions}
1. If the pattern $p$ has bound variables $x_1 \commadots x_n$, where $n > 1$:
-val $\Dollar x$ = $e$ match {case $p$ => {$x_1 \commadots x_n$}}
+val $\Dollar x$ = $e$ match {case $p$ => ($x_1 \commadots x_n$)}
val $x_1$ = $\Dollar x$._1
val $x_n$ = $\Dollar x$._n .
@@ -1959,7 +1951,7 @@ \section{Value Declarations and Definitions}
val x = f() match { case Some(x) => x }
-val x$\Dollar$ = mylist match { case x :: xs => {x, xs} }
+val x$\Dollar$ = mylist match { case x :: xs => (x, xs) }
val x = x$\Dollar$._1
val xs = x$\Dollar$._2
@@ -2029,7 +2021,7 @@ \section{Variable Declarations and Definitions}
\lstinline@0.0f@ & if $T$ is \code{Float},\\
\lstinline@0.0d@ & if $T$ is \code{Double},\\
\code{false} & if $T$ is \code{Boolean},\\
-\lstinline@{}@ & if $T$ is \code{Unit}, \\
+\lstinline@()@ & if $T$ is \code{Unit}, \\
\code{null} & for all other types $T$.
When they occur as members of a template, both forms of variable
@@ -2285,7 +2277,7 @@ \section{Variance Annotations}\label{sec:variances}
the variance position of the enclosing type ~\lstinline@$S$[$\ldots T \ldots$ ]@.
\todo{handle type aliases}
-References to the type parameters in object-private values, variables,
+References to the type parameters in object-private or object-protected values, variables,
or methods (\sref{sec:modifiers}) of the class are not checked for their variance
position. In these members the type parameter may appear anywhere
without restricting its legal variance annotations.
@@ -2519,7 +2511,7 @@ \subsection{Procedures}\label{sec:procedures}
Special syntax exists for procedures, i.e.\ functions that return the
-\verb@Unit@ value \verb@{}@.
+\verb@Unit@ value \verb@()@.
A procedure declaration is a function declaration where the result type
is omitted. The result type is then implicitly completed to the
\verb@Unit@ type. E.g., ~\lstinline@def $f$($\ps$)@~ is equivalent to
@@ -2769,17 +2761,6 @@ \section{Templates}
the first parent class of a template is a regular superclass
constructor, not a trait reference.
-The list of parents of every class is also always implicitly extended
-by a reference to the \code{scala.ScalaObject} trait as last
-mixin. E.g.\
-$sc$ with $mt_1$ with $\ldots$ with $mt_n$ {$\stats\,$}
-$mt_1$ with $\ldots$ with $mt_n$ with ScalaObject {$\stats\,$}.
The list of parents of a template must be well-formed. This means that
the class denoted by the superclass constructor $sc$ must be a
subclass of the superclasses of all the traits $mt_1 \commadots mt_n$.
@@ -2947,10 +2928,8 @@ \subsection{Class Linearization}\label{sec:linearization}
Then the linearization of class \lstinline@Iter@ is
-{ Iter, RichIterator, StringIterator, AbsIterator, ScalaObject, AnyRef, Any }
+{ Iter, RichIterator, StringIterator, AbsIterator, AnyRef, Any }
-Trait \lstinline@ScalaObject@ appears in this list because it
-is added as last mixin to every Scala class (\sref{sec:templates}).
Note that the linearization of a class refines the inheritance
relation: if $C$ is a subclass of $D$, then $C$ precedes $D$ in any
@@ -2960,13 +2939,13 @@ \subsection{Class Linearization}\label{sec:linearization}
as a suffix. For instance, the linearization of
\lstinline@StringIterator@ is
-{ StringIterator, AbsIterator, ScalaObject, AnyRef, Any }
+{ StringIterator, AbsIterator, AnyRef, Any }
which is a suffix of the linearization of its subclass \lstinline@Iter@.
The same is not true for the linearization of mixins.
For instance, the linearization of \lstinline@RichIterator@ is
-{ RichIterator, AbsIterator, ScalaObject, AnyRef, Any }
+{ RichIterator, AbsIterator, AnyRef, Any }
which is not a suffix of the linearization of \lstinline@Iter@.
@@ -3279,7 +3258,7 @@ \section{Modifiers}
templates inside $C$.
An different form of qualification is \code{private[this]}. A member
-$M$ marked with this modifier can be accessed only from within
+$M$ marked with this modifier is called {\em object-protected}; it can be accessed only from within
the object in which it is defined. That is, a selection $p.M$ is only
legal if the prefix is \code{this} or \lstinline@$O$.this@, for some
class $O$ enclosing the reference. In addition, the restrictions for
@@ -4036,9 +4015,10 @@ \section{The {\em Null} Value}
\lstinline@isInstanceOf[$T\,$]@ always returns \code{false}.
-\lstinline@asInstanceOf[$T\,$]@ returns the ``null'' object itself if
-$T$ conforms to \lstinline@scala.AnyRef@, and throws a
-\lstinline@NullPointerException@ otherwise.
+\lstinline@asInstanceOf[$T\,$]@ returns the default value of that type, as
+specified in \sref{sec:vardef}.
+\lstinline@##@ returns \lstinline@0@.
A reference to any other member of the ``null'' object causes a
\code{NullPointerException} to be thrown.
@@ -4912,7 +4892,7 @@ \section{For Comprehensions and For Loops}\label{sec:for-comprehensions}
Enumerators ::= Generator {semi Enumerator}
Enumerator ::= Generator
| Guard
- | `val' Pattern1 `=' Expr
+ | Pattern1 `=' Expr
Generator ::= Pattern1 `<-' Expr [Guard]
Guard ::= `if' PostfixExpr
@@ -5409,7 +5389,7 @@ \subsection{Value Conversions}
\paragraph{\em View Application}
If none of the previous conversions applies, and $e$'s type
does not conform to the expected type $\proto$, it is attempted to convert
-$e$ to the expected type with a view (\sref{sec:views}).\bigskip
+$e$ to the expected type with a view (\sref{sec:views}).
\paragraph{\em Dynamic Member Selection}
If none of the previous conversions applies, and $e$ is a prefix
@@ -5777,25 +5757,7 @@ \subsection{Eta Expansion}\label{sec:eta-expand}
\subsection{Dynamic Member Selection}\label{sec:dyn-mem-sel}
-The standard Scala library defines a trait \lstinline@scala.Dynamic@ which defines a member
-\@invokeDynamic@ as follows:
-package scala
-trait Dynamic {
- def applyDynamic (name: String, args: Any*): Any
- ...
-Assume a selection of the form $e.x$ where the type of $e$ conforms to \lstinline@scala.Dynamic@.
-Further assuming the selection is not followed by any function arguments, such an expression can be rewitten under the conditions given in \sref{sec:impl-conv} to:
-If the selection is followed by some arguments, e.g.\ $e.x(\args)$, then that expression
-is rewritten to
-$e$.applyDynamic("$x$", $\args$)

This doesn't seem mentioned in the issue description, and Scala does have scala.Dynamic (with some slightly different details). So I'm against merging this bit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
\chapter{Implicit Parameters and Views}\label{sec:implicits}
@@ -6726,11 +6688,7 @@ \section{Type Patterns}\label{sec:type-patterns}
flag the possible loss of type-safety.
A {\em type variable pattern} is a simple identifier which starts with
-a lower case letter. However, the predefined primitive type aliases
-\lstinline@unit@, \lstinline@boolean@, \lstinline@byte@,
-\lstinline@short@, \lstinline@char@, \lstinline@int@,
-\lstinline@long@, \lstinline@float@, and \lstinline@double@ are not
-classified as type variable patterns.
+a lower case letter.
\section{Type Parameter Inference in Patterns}\label{sec:type-param-inf-pat}
@@ -7229,10 +7187,10 @@ \section{Programs}
would work as well.
\code{HelloWorld} can also be defined without a \code{main} method
-by inheriting from \code{Application} instead:
+by inheriting from \code{App} instead:
package test
-object HelloWord extends Application {
+object HelloWord extends App {
println("hello world")
@@ -7583,16 +7541,13 @@ \section{Root Classes}
subclasses: \code{AnyRef} and \code{AnyVal}.
The subclass \code{AnyRef} represents all values which are represented
-as objects in the underlying host system. Every user-defined Scala
-class inherits directly or indirectly from this class. Furthermore,
-every user-defined Scala class also inherits the trait
-\code{scala.ScalaObject}. Classes written in other languages still
-inherit from \code{scala.AnyRef}, but not from
+as objects in the underlying host system. A user-defined Scala class
+inherits directly or indirectly from this class (except when inheriting
+from \code{AnyVal} explicitly). Classes written in other languages
+inherit from \code{scala.AnyRef}.
-The class \code{AnyVal} has a fixed number of subclasses, which describe
-values which are not implemented as objects in the underlying host
+Subclasses of \code{AnyVal} describe values which are not implemented
+as objects in the underlying host system.
Classes \code{AnyRef} and \code{AnyVal} are required to provide only
the members declared in class \code{Any}, but implementations may add
@@ -7650,8 +7605,6 @@ \section{Root Classes}
def synchronized[T](body: => T): T // execute `body` in while locking `this`.
-/** A mixin class for every user-defined Scala class */
-trait ScalaObject extends AnyRef
The type test \lstinline@$x$.isInstanceOf[$T$]@ is equivalent to a typed
@@ -7930,7 +7883,7 @@ \subsection{The \large{\code{Tuple}} classes}
package scala
-case class Tuple$n$[+a_1, ..., +a_n](_1: a_1, ..., _$n$: a_$n$) {
+case class Tuple$n$[+A_1, ..., +A_n](_1: A_1, ..., _$n$: A_$n$) {
def toString = "(" ++ _1 ++ "," ++ $\ldots$ ++ "," ++ _$n$ ++ ")"
@@ -7947,8 +7900,8 @@ \subsection{The \large{\code{Function}} Classes}
package scala
-trait Function$n$[-a_1, ..., -a_$n$, +b] {
- def apply(x_1: a_1, ..., x_$n$: a_$n$): b
+trait Function$n$[-A_1, ..., -A_$n$, +B] {
+ def apply(x_1: A_1, ..., x_$n$: A_$n$): B
def toString = "<function>"
10 documentation/src/reference/ScalaReference.tex
@@ -66,7 +66,7 @@
\renewcommand{\doctitle}{Scala By Example\\[33mm]\ }
\renewcommand{\docauthor}{Martin Odersky\\[53mm]\ }
\renewcommand{\doctitle}{The Scala Language \\ Specification \\
-\Large Version 2.8 \ }
+\Large Version 2.10 \ }
\renewcommand{\docauthor}{Martin Odersky \\
@@ -89,14 +89,6 @@
-%\todo{`:' as synonym for $\EXTENDS$?}
-%\part{The Scala Language Specification}
8 documentation/src/reference/SyntaxSummary.tex
@@ -24,16 +24,14 @@ \chapter{Scala Syntax Summary}\label{sec:syntax}
| `\`' stringLit `\`'
idrest ::= {letter | digit} [`_' op]
- integerLiteral ::= (decimalNumeral | hexNumeral | octalNumeral) [`L' | `l']
+ integerLiteral ::= (decimalNumeral | hexNumeral) [`L' | `l']
decimalNumeral ::= `0' | nonZeroDigit {digit}
hexNumeral ::= `0' `x' hexDigit {hexDigit}
- octalNumeral ::= `0' octalDigit {octalDigit}
digit ::= `0' | nonZeroDigit
nonZeroDigit ::= `1' | $\cdots$ | `9'
- octalDigit ::= `0' | $\cdots$ | `7'
- ::= digit {digit} `.' {digit} [exponentPart] [floatType]
+ ::= digit {digit} `.' digit {digit} [exponentPart] [floatType]
| `.' digit {digit} [exponentPart] [floatType]
| digit {digit} exponentPart [floatType]
| digit {digit} [exponentPart] floatType
@@ -160,7 +158,7 @@ \chapter{Scala Syntax Summary}\label{sec:syntax}
Enumerators ::= Generator {semi Enumerator}
Enumerator ::= Generator
| Guard
- | `val' Pattern1 `=' Expr
+ | Pattern1 `=' Expr
Generator ::= Pattern1 `<-' Expr [Guard]
CaseClauses ::= CaseClause { CaseClause }
Something went wrong with that request. Please try again.