pandas.DataFrame.tail #343
Comments
|
Is the most common case people run tail after sorting? It's actually somewhat poor performance to implement a general purpose tail. |
|
It is not as straightforward as it seems. In Pandas, all the data is already in memory, |
|
spark tends to think of tail as what happened in a linked list (dropping just a few elements), but it is effectively used like .head or .limit on the end of the data. I have started a small design doc on how we could support these use cases and at the same time ensure that the output is exactly the same as pandas despite the distributed nature of spark. I will share it this weekend or next week. |
|
If you run a large data set in pandas, it may be a good idea to run both head and tail methods to confirm the integrity of your data while uploading or after processing your data. |
|
Any update on this? For the demo, I'd actually really like tail to be implemented. |
|
^ WDYT @rxin, @thunterdb and @ueshin? |
### What changes were proposed in this pull request? This PR proposes a `tail` API. Namely, as below: ```scala scala> spark.range(10).head(5) res1: Array[Long] = Array(0, 1, 2, 3, 4) scala> spark.range(10).tail(5) res2: Array[Long] = Array(5, 6, 7, 8, 9) ``` Implementation details will be similar with `head` but it will be reversed: 1. Run the job against the last partition and collect rows. If this is enough, return as is. 2. If this is not enough, calculate the number of partitions to select more based upon `spark.sql.limit.scaleUpFactor` 3. Run more jobs against more partitions (in a reversed order compared to head) as many as the number calculated from 2. 4. Go to 2. **Note that**, we don't guarantee the natural order in DataFrame in general - there are cases when it's deterministic and when it's not. We probably should write down this as a caveat separately. ### Why are the changes needed? Many other systems support the way to take data from the end, for instance, pandas[1] and Python[2][3]. Scala collections APIs also have head and tail On the other hand, in Spark, we only provide a way to take data from the start (e.g., DataFrame.head). This has been requested multiple times here and there in Spark user mailing list[4], StackOverFlow[5][6], JIRA[7] and other third party projects such as Koalas[8]. In addition, this missing API seems explicitly mentioned in comparison to another system[9] time to time. It seems we're missing non-trivial use case in Spark and this motivated me to propose this API. [1] https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.tail.html?highlight=tail#pandas.DataFrame.tail [2] https://stackoverflow.com/questions/10532473/head-and-tail-in-one-line [3] https://stackoverflow.com/questions/646644/how-to-get-last-items-of-a-list-in-python [4] http://apache-spark-user-list.1001560.n3.nabble.com/RDD-tail-td4217.html [5] https://stackoverflow.com/questions/39544796/how-to-select-last-row-and-also-how-to-access-pyspark-dataframe-by-index [6] https://stackoverflow.com/questions/45406762/how-to-get-the-last-row-from-dataframe [7] https://issues.apache.org/jira/browse/SPARK-26433 [8] databricks/koalas#343 [9] https://medium.com/chris_bour/6-differences-between-pandas-and-spark-dataframes-1380cec394d2 ### Does this PR introduce any user-facing change? No, (new API) ### How was this patch tested? Unit tests were added and manually tested. Closes #26809 from HyukjinKwon/wip-tail. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
As since Spark 3.0 supports a `tail()`, this PR proposes a `Series.tail()` and `DataFrame.tail()` for Koalas based on it.
- Series
```python
>>> kser = ks.Series([1, 2, 3, 4, 5])
>>> kser
0 1
1 2
2 3
3 4
4 5
Name: 0, dtype: int64
>>> kser.tail(3)
2 3
3 4
4 5
Name: 0, dtype: int64
```
- DataFrame
```python
>>> df = ks.DataFrame({'animal': ['alligator', 'bee', 'falcon', 'lion',
... 'monkey', 'parrot', 'shark', 'whale', 'zebra']})
>>> df
animal
0 alligator
1 bee
2 falcon
3 lion
4 monkey
5 parrot
6 shark
7 whale
8 zebra
>>> df.tail()
animal
4 monkey
5 parrot
6 shark
7 whale
8 zebra
>>> df.tail(3)
animal
6 shark
7 whale
8 zebra
>>> df.tail(-3)
animal
3 lion
4 monkey
5 parrot
6 shark
7 whale
8 zebra
```
Resolves #343
Head is implemented but tail is not. Tail is not just useful for exploring data but also a very useful way of slicing data.
Example: users have data ordered by date. They want to take the last 30 days. They can just call tail(30)
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.tail.html
The text was updated successfully, but these errors were encountered: